Chapter 37. Operations in Evaluating Community Interventions | Community Tool Box

• What do we mean by choosing evaluation questions?

• Why is it necessary to choose evaluation questions carefully?

• When should you choose questions and plan the evaluation?

• Who should be involved in the process?

• How do you choose questions and plan the evaluation?

Chapters 36-39 of the Community Tool Box concern the evaluation of community programs.  We've chosen to devote so much space to evaluation because it's one of the most important parts of any effort to improve community life and bring about lasting social change.  It can help you to better understand how well the planning and preparations for your program went, whether you implemented it as you meant to, and what the consequences were.  It can tell you whether you met your original objectives and goals or not, and give you information about what you need to change to be more effective.

Chapter 36 explains how an evaluation works, and gives some guidance as to how to develop it. Chapter 38 will assist you in gathering and analyzing the information you want, and Chapter 39 deals with how to use that information to improve your program and garner community and funding support. Here, in Chapter 37, we look at evaluation from a research point of view - how to plan and structure an evaluation that can help you better understand and improve what you do.

In this section, we'll discuss the first, and perhaps most important, step in evaluation research: deciding exactly what to evaluate. Each of the rest of the sections in the chapter will deal in detail with one of the steps you'll need to take to design, implement, and use the evaluation. The goal of the chapter is to provide guidelines that are useful to grassroots or community-based organizations as well as students or academic researchers.

What do we mean by choosing questions, and why is it necessary?

Every evaluation, like any other research, starts with one or more questions. Sometimes, the questions are simple and easy to answer. (Will we serve something close to the 50 people we expect to?) Often, however, the questions can be complex, and the answers less easy to find. (Which, or which combination, of the three parts of our intervention will affect which of the two behavior changes we seek within participants?) The questions you ask will guide not only your evaluation, but your program as well. By your choice of questions, you're defining what it is you're trying to change.

You choose your evaluation questions by analyzing the community problem or issue you're addressing, and deciding how you want to affect it. Why do you want to ask this particular question in relation to your evaluation? What is it about the issue that is the most pressing to change? What indicators will tell you whether that change is taking place?  Is that all you're concerned with? The answer to each of these and other questions helps to define what it is you're trying to do, and, by extension, how you'll try to do it.

For example, what's the real goal of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss in kids who are overweight or obese.  It could be to educate them about healthy eating, and to persuade them to be more adventurous eaters.

The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure weight loss, for instance, then clearly that's not what you're aiming for. If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods? You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

Things to consider when choosing evaluation questions

What do you want to know?

Academics and other researchers may approach choosing research questions differently from those involved in community programs.  In addition to their practical and social applications, they may choose problems to research simply because they are interesting, or because they tie into other work that they or their colleagues are doing. Community service workers and others directly involved in programs, on the other hand, are concerned specifically with improving what they're doing so they can help to enhance the quality of life for the participants in their programs, and often for the community as a whole.  Since we assume that most people using this chapter of the Tool Box are likely to be practitioners in the community, let's look at some of the reasons they might pick a particular area to evaluate.

If you're running, or about to run, a program to affect a community issue or problem, you might want to know one or more of the following:

Is there a cause-and-effect relationship (i.e., does one action or condition directly cause another) between a particular action and a particular change?  Usually, you'll be concerned with this in terms of your program. (Does our smoking-cessation support group help members to quit smoking?) Sometimes, however, it might be important to look at it in terms of the community. (Does a smoking ban in public buildings, bars, and restaurants lead to a decrease in the number of community residents who smoke?)

If we try this new method, what will happen?

Will the program that worked in the next town, or the one that we read about in a professional journal, work with our population, or with our issue?

Why are you interested?

Some of the same differences between the concerns of researchers and the concerns of practitioners may hold here. Those interested primarily in research may simply be moved by curiosity or by the urge to solve a difficult problem. As a practitioner, on the other hand, you'll want to know the effects of what you're doing on the lives of participants or the community.

Your interest, therefore, might grow from:

• Your experience with an issue and its consequences in a particular population or community
• Your knowledge of promising interventions and their effects on similar issues
• The uniqueness of the issue to your particular community or population
• The similarity of the issue to other issues in your community, or the issue's interaction with other issues

Your interest as a community worker has to be considered in relation to your evaluation and the purpose of your program. Your basic intent is probably to improve things for the population or the community, but in what ways and by what means? Are you trying out some new things in the hope of making an already-successful program more successful? Are you importing a promising practice to see if it works with your population?  Are you trying to solve a particularly difficult professional problem?

A community mediation program found that it was having little success in cases involving adolescents.  After conferring with other similar programs - all of which were struggling with the same issue - mediators in the program devised a number of strategies to try to reach youth. The overall question they were concerned with - "Will these strategies make it possible to mediate successfully where teens are involved?" - was one with real consequences.

Is the issue you're addressing important to the community or to the society?

Media reports about or community attempts to address the issue are clear indicators that it is socially important. If it affects a particular group - violence in a given neighborhood, a high rate of heart disease among middle-aged Black males - it has an obvious impact on the community and society. If your program or intervention has the potential to help resolve the issue in other places, to be used by community workers in other fields, or to be applied in a number of ways, the importance of your analysis increases even further. If addressing the issue can lead to long-term positive social change, then the analysis is vitally important.

All of this affects your evaluation and the questions you ask. If the issue is one of social importance, then your evaluation of your work is socially important as well.  Are you addressing the aspects of your program or intervention that are of the greatest value to participants, the community, and society?  If not, how might you begin to do so?

How does the issue relate to the field?

The real question here is not whether the issue is important to the field - if it's important to the community, that's what matters. However, you should explore whether there's evidence from the field to apply to the issue.  Is what you're doing likely to be more effective than other approaches that have been tried?  If your approach isn't effective, are there other approaches out there that hold more promise? Can the published material about the issue help you understand it better, and give you better ideas about how to address it?

Is the issue general, rather than specific to your population or community?

Consider whether there is evidence that the issue occurs with a variety of populations and under a range of conditions. Also consider whether the observations or methods used to determine the issue's existence are accurate and whether they can be used in different situations and with different groups. Your evaluation may give you valuable information to pass on to practitioners in different fields or different circumstances.

Who might use the results of your evaluation?

If evaluation shows that your program or intervention is successful, that's obviously valuable information, especially if what you're evaluating is innovative and hasn't been tried before. Even if the evaluation turns up major problems with the intervention, that's still important information for others - it tells them what won't work, or what barriers have to be overcome in order to make it work.

Some of those who might use your results include individuals and groups affected by the issue; service providers and others who have to deal with the problem (in the case of youth violence, for instance, this last group might include police, school officials, small business owners, parents, and medical personnel, among others); advocates and community activists; and public officials and other policy makers.

Whose issue is it?

Who has to change in order to address the issue? The focus of the intervention will tell you whom the evaluation should focus on.

Some possibilities:

• Those directly affected by the problem
• Those in direct personal contact with those directly affected: parents, spouses and children, other relatives, friends, neighbors, coworkers
• Those who serve or otherwise deal with those directly affected: medical professionals, police, teachers, social workers, therapists, etc.
• Administrators and others who serve or deal with those indirectly affected: hospital or clinic directors, police chiefs, school principals, agency directors, etc.
• Appointed or elected officials and other policy makers

Why is it necessary to choose evaluation questions carefully?

You know why you're running your program.  Evaluating it should just be a matter of deciding whether things are better when you evaluate than they were before you started, right?  Well, actually...wrong.  It's not that simple. First of all, you need to determine what "things" you are actually looking at (remember the school lunch example?)  Second, you will need to consider how you will determine what you're doing right, and what you need to change. Here's a partial list of reasons why choosing questions beforehand is important.

• It helps you understand what effects different parts of your effort are having. By framing questions carefully, you can evaluate different parts of your effort. If you add an element after the start of the program, for instance, you may be able to see its effect separate from that of the rest of the program...if you focus on examining it.  By the same token, you can look at different possible effects of the program as a whole. (Do adult basic education learners read more as a result of being in a program?  Are they more likely to register to vote? Do their children improve their school performance?)
• It makes you clearly define what it is you're trying to do  What you decide to evaluate defines what you hope to accomplish. Choosing evaluation questions at the start of a program or effort makes clear what you're trying to change, and what you want your results to be.
• It shows you where you need to make changes. Carefully choosing questions and making them specific to your real objectives should tell you exactly where the program is doing well and where the program isn't having the intended effect.
• It highlights unintended consequences. When you find unusual answers to the questions you choose, it often means that your program has had some effects you didn't expect. Sometimes these effects are positive - not only did people in the heart-healthy exercise program gain in fitness, but a majority of them report changing their diet for the better and losing weight as well - sometimes negative - obese children in a healthy eating program actually gained weight, even though they were eating a healthier diet - and sometimes neither. Like the side effects of medication, the unintended consequences of a program can be as important as the program itself. (In the case of the exercise program, the changes in diet might do as much as or more than the exercise to maintain heart health, for instance, and may point toward changing the focus of the program in some way.)
• It guides your future choices. If you find that your program is particularly successful in certain ways and not in others, for example, you may decide to emphasize the successful areas more, or to completely change your approach in the unsuccessful areas. That, in turn, will change the emphasis of future evaluation as well.
• In participant evaluations, evaluation involves stakeholders in setting the course of the program, thus making it more likely that it will meet community needs.
• It provides focus for the evaluation and the program. Choosing evaluation questions carefully keeps you from becoming scattered and trying to do too many things at once, thereby diluting your effectiveness at all of them.
• It determines what needs to be recorded in order to gather data for evaluation. A clear choice of evaluation questions makes the actual gathering of data much easier, since it usually makes obvious what kinds of records must be kept and what areas need to be examined.

When should you choose questions and plan the evaluation?

Evaluation questions, since they help shape your work, should be chosen and the evaluation planned when planning the overall program or effort. That gives you time and room for a participatory process, and gives you the chance to use the evaluation as an integral part of the program. As the program unfolds, you might find yourself adjusting or adding questions to reflect the reality of what is happening, but unless your original questions were misguided (you were wrong about what behavior had to change in order to produce certain results, for instance), they should serve you well.

Now let's discuss reality for many community based and grassroots programs. They're often understaffed and underfunded. Staff members may be underpaid, and may often work many more hours a week than they're paid for, because of their dedication to social justice and social change. Most or all program staff may even be volunteers, with full-time jobs and family responsibilities aside from their work in the program.  Initial evaluation in these circumstances is often anecdotal - i.e., based on participants' comments and stories about their progress and staff members' personal, informal observations.  A formal evaluation will probably wait until there's funding for it, or until someone has the time to coordinate or take charge of it.

In that case, the "when" becomes "as soon as you can."  You may be dealing with a program that has just started, or with one that's been operating for a long time. You may know that changes need to be made, or it may seem that the program is in fact meeting its goals. Whatever the situation, evaluation questions need to be chosen, and an evaluation planned that will give you the information you need to improve your work. Even with a program that's been going on for a while, the questions can still help you define or redefine your work, and will certainly help you improve it over the long term.

Who should be involved in choosing questions and planning the evaluation?

If you've consulted other sections of the Tool Box concerned with evaluation, you probably know that we advocate that all stakeholders be involved in planning the evaluation.  We believe that the best evaluation is participatory. That means that there is representation of the views and knowledge of people affected by the issue to be addressed. The list of potential participants is essentially the same as that under "Whose problem is it?" in the first part of this section: those directly affected and their close contacts; those who work with those directly affected, or who deal directly or indirectly with them and the issue; and public officials. To these groups, we might add other concerned citizens, and those indirectly affected by the issue. (A shop owner may not be a victim of neighborhood violence, but fear of that violence might nonetheless keep customers away from his shop, for instance.)

Evaluations that involve all stakeholders have a number of advantages over those conducted in a vacuum by outside evaluators or agency or program staff. They're more likely to reflect the real needs of the community, and they bring to bear the community's knowledge of its own context - history, relationships, culture, etc. - without which a program and its evaluation can go astray.

Participation can range from simple consultation before the fact to complete involvement in every aspect of an evaluation - assessment, planning, data gathering, analysis, and passing on the information.  In general, the greater the involvement of stakeholders, the better, but in-depth involvement of the stakeholders may not always be possible. There are time disadvantages to participatory evaluation - it takes longer - and there are logistical concerns, as well. Participants may have nothing in their backgrounds to prepare them for research, so training in a number of areas may be necessary, requiring skill, careful planning, and yet more time. The level of participation your evaluation can sustain, therefore, relies to some extent on your time constraints and your capacity to train and support participants.

How do you choose questions and plan the evaluation?

Choosing questions

When you choose evaluation questions, you're really choosing a research problem - what you want to examine with your research. (Evaluation, whether formal or informal, is in fact research.) You have to analyze the issue and your program, consider various ways they can be looked at, and choose the one(s) that most nearly tell you what you want to know about what you're doing.  Are you just trying to determine whether you're reaching the right people in sufficient numbers with your program?  Do you want to know how well an intervention is working with specific populations? What kinds of behavior changes, if any, are taking place as a result?  What the actual outcomes are for the community? Each of these - as well as each of the many other things you might want to know - implies a different set of evaluation questions. To find the questions that best suit your evaluation, there is a series of steps you can follow.

Describe the issue or problem you're addressing

A problem is a difference between some ideal condition (all people 10 years of age or older should be able to read; people should be able to find a decent job) and some actual condition in the community or society (a 25% illiteracy rate among those attending a particular high school; 50% unemployment among minority youths in a particular city). This may mean the absence of some positive factor (qualified teachers and adequate educational facilities; entry-level jobs that are reachable from minority neighborhoods) or the presence of some negative factor (students' difficulty with English; discrimination against minority job applicants), or some combination of these.

To describe the issue or problem:

• Describe the ideal condition, including the positive factors present and the negative factors absent.  What should it look like if everything was as you'd want it to be?
• Describe the actual conditions that constitute the prolem of interest, including the negative conditions present and the positive conditions absent.  What are conditions really like?
• Describe the actual problem in terms of what you're hoping to change.  What positive factors do you want to produce and/or what negative factors do you want to eliminate?

Describe the importance of the problem

To be sure that this is a problem you really should be addressing, consider its importance to those affected and to the community.

• Is the discrepancy between ideal and actual conditions of the kind and size to be considered important?
• What are the consequences (positive and negative) of the problem?
• Who experiences these consequences (i.e. program participants; their families, friends, and peers; service providers, policymakers, and others)? How many people are affected?
• How often and for how long are they affected? What is the intensity of the effect?
• How much does the fact that the problem is experienced to this degree by these people matter to them?

You might also ask whether the effects of the problem matter to society, but in fact, that shouldn't make a difference.  If they matter to the people who experience them, they're important.  Society doesn't always consider a problem important if it's only a problem for a minority, or for a group that's generally ignored (the poor, the homeless).

In light of these factors, decide whether the problem is important to the evaluation.

Describe those who contribute to the problem

Whose behavior, by its presence or absence, contributes to the problem?  Are they in the program participants' personal environment (participants themselves, family, friends), service environment (teachers, police), or broader environment (policymakers, media, general public)? For each of them, consider the types of behavior that, by their presence or absence, contribute to the discrepancy that constitutes the problem.

Assess the importance and feasibility of changing those behaviors

How important is each of these behaviors to solving the problem?  What are the chances that your effort can have any effect on each of them?

Describe the change objective

Based on the above analysis, choose behavior changes to target in specific people. Where you can, specify the desired levels of change in targeted behaviors and outcomes (those changes in conditions that should occur if the problem were to be solved).

For example, a behavior change goal might be an increase in pre-employment capacity - self-presentation, job-seeking, interview skills, interpersonal competence, resume writing, basic skills, etc. - for minority job seekers aged 18-24. Or you might instead or in addition target policy makers, with the goal of having them offer tax incentives to businesses that locate in or close to minority communities.

This is a way of defining your work. If you're planning the evaluation as you plan the program - as you would in the ideal situation - then the questions you're asking the evaluation to examine reflect the problems you're trying to solve, and this kind of analysis is important.  If you're starting an evaluation of a program that has been in place for some time, then you're going to have to do some figuring after the fact about what consequences you think (hope) the program is having, and what they will lead to.  You may be talking about changes in specific participant behaviors, about behaviors that act as indicators of other changes, or about results of another sort (participants gaining employment, for instance, which may have a direct relationship to participant behavior or may have more to do with local economic conditions).

Make sure that the expected changes would constitute a solution or substantial contribution to the problem

If you conclude that they would not result in a substantial contribution, revise your choice of problem and/or your selection of targeted people and actions as necessary. If you think that what you're looking at in an evaluation doesn't address the problem, then you should be looking at something else. If the objectives you've chosen do constitute all or a substantial part of a solution, you've found your questions.

Setting

Now that you've chosen your questions, there may be other factors to consider, such as the settings in which the evaluation will be conducted. If your program is relatively small and/or has only one site, this wouldn't be an issue. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program.

 There are some situations in which the choice setting may be important:

• If your program is very large and/or has multiple sites
• If different sites provide different services, activities, or conditions, or use different methods

Multiple sites

Multiple sites

Can present a challenge for an evaluation, because, although every effort may be made to make the program at all sites exactly the same, it will seldom be so. If the program relies on human interaction - teacher/learner, counselor/counselee, trainer/trainee, doctor/patient, etc. - there will be differences from site to site depending on the people staffing each. (The exception is when the same people staff all sites, providing the same services at each site at different times or on different days.)  Even if all are equally competent, no two staff members or teams will do things in exactly the same way or relate to participants in exactly the same way, and the differences can be reflected in differences in outcomes.  If methods or other factors vary from site to site, that will further complicate the situation.

Furthermore, the physical character of a site can influence not only program effectiveness, but also the recruitment of participants and whether or not they remain in the program long enough for it to have some effect (often called "retention.")  The site's layout, comfort, apparent safety and security, and - often most important - how easy it is to get to, all affect whether participants enroll and stay in the program.

Where you do have the capacity to evaluate all sites, it will be helpful to build into the evaluation a method of comparing them. This will allow you to identify and adopt at all sites methods, conditions, or activities that seem to make one site particularly successful, and to identify and change at all sites methods, conditions, or activities that seem to create barriers to success at others.

If you can't evaluate each site separately, you'll have to decide which one(s) will give you the information that will most help in adjusting and improving your program. If you're most concerned with assessing your overall effectiveness, this may mean evaluating the site(s) closest to the program norm, in terms of methods, conditions, activities, goals, participant/staff interaction, etc. If, on the other hand, your chief consideration is learning whether a particular new or unusual method or situation is working, you may find yourself evaluating the site(s) least like the others.

If sites appear only minimally different, some other considerations that may come into play are:

• The number and character of participants at the site. Participants at a particular site may be experiencing the effects of the issue more severely, or may have a particular important characteristic, such as a language barrier.
• The ability and willingness of participants and staff to support the evaluation research. If staff at a particular site are unable or unwilling to record observations, attendance, and other key information, or if site participants are unable or unwilling to be interviewed or monitored, evaluation at that site might be difficult.
• The stability of the population at the site. If participants at a site come and go at a rapid rate - unless that's the program's intent - it can be difficult to gain information that contributes to an accurate evaluation.
• An exception, of course, occurs here if one point of the evaluation is to find out why participants stay for so short a time, and to try to develop methods or create conditions to assist them to remain in the program long enough to reach their goals.

Sites with different methods, conditions, activities, or services

Programs sometimes are organized so that different methods are used or different services provided at different sites. In other cases, conditions may vary from site to site because of the sites' geographical locations or the available space. The ideal situation is to evaluate all sites and compare the effects of the different methods, conditions, or services. When that's not possible, you'll have to decide what's most important to find out.

If the methods, services, or conditions at a particular site are new or innovative, you may want to evaluate them, rather than those that have a track record. There may be a particular method or service that you want to evaluate, in which case the decision about which site to choose is obvious. The decision should be based on what makes the most sense for your program, and what will give you the best information to improve its effectiveness.

When you have the capacity to choose more than one site to evaluate, it often makes sense to choose two or three sites that are different - especially if each is representative of other sites in the program or of program initiatives - so that you can compare their effectiveness. Even where sites are essentially similar, you'll get more information by evaluating as many as you can.

Participants

Another factor to consider is the participants whose behavior, activity, or circumstances will be evaluated. If your program is relatively small this might not be an issue - the participants will simply be all those in the program. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program, there are some situations in which the choice of participants may be important:

• If your program includes different groups of participants (groups that are in different stages of the program, or that are exposed to different methods or services).
• If groups of participants belong to populations with distinctly different cultures, stemming from race, ethnicity, class, religion, or other factors.

Multiple groups

There are a number of reasons why there might be multiple groups of participants in a program. You might start different groups at different times, either because the program has a rolling start schedule (when there are enough people for a class/training group, one will begin), or because the program is aimed at different groups (for example, 5 year-olds, 8-year-olds, and 14-year-olds). You might also be trying different strategies with different groups.

The Brookline Early Education Project (BEEP), a program aimed at school readiness for children aged pre-birth through 5, recruited pregnant families in three cohorts over the course of three years.  In addition, families in each cohort were assigned to one of three levels of service. Thus, there were actually nine different groups among BEEP participants, even though, by the third year, all were receiving services at the same time.

Once again, if there's no problem in evaluating the whole program, participants will simply include everyone. If that's not possible, there are a number of potential choices:

Evaluate your work with only one group, with the expectation that work with the others will be evaluated in the future. In this case, you'd probably want to choose the one for whom you consider the program most crucial. They might be at greater risk (of heart attack, of school failure, of homelessness, etc.) or might be experiencing the issue at a high level of intensity (daily shooting incidents in the neighborhood, high rates of teen pregnancy, massive unemployment).

Include a small number (2-4) of groups in your evaluation. You might want to choose groups with contrasting characteristics (different ages, for example, or addressed by different strategies). On the other hand, depending on the focus of your evaluation, you might want groups that are essentially similar, to see whether your work is consistent in its effects.

Choose a few participants from each group to focus your evaluation on.  While this won't give you a complete picture, it should give you enough information to tell where your program is accomplishing its goals and where it needs improvement. The differences in the ways participants in different groups respond to the program (assuming there are differences) can also give you ideas for ways to change what you're doing.

Participants from different populations and cultures

Cultural factors can have an enormous effect on participants' responses to a program. They can govern conceptions of social roles, family responsibilities, acceptable and unacceptable behavior, attitudes toward authority (and who constitutes authority), allowable topics of conversation, morality, the role of religion - the list goes on and on. In planning a program that involves members of different populations and cultures, you essentially have three choices:

• Plan your program and implement it in the same way for everyone. If the program involves groups - classes, support groups, etc. - participants' membership is determined not by population group but by when they sign up, what time of day they can attend, what they sign up for, or whatever other criteria make sense logistically.
• Plan your program to be as culturally sensitive as possible, and try to screen out anything that might be offensive to or difficult for any group. In this instance, you might be prepared to respond if participants from a particular population requested a group of their own.
• Divide participants by cultural group and plan different culturally sensitive approaches for each. Your overall approach might be the same for everyone, but the way you apply it might differ by culture.

In any of these instances, it would probably be important to understand how well your approach is working with members of the various populations.  If you can evaluate the whole program, make sure that you include enough members of each group so that you can compare results (and their opinions of the program) among them.  If your evaluation possibilities are limited, then your choices are similar to those for multiple groups of other kinds, and will depend on what exactly is most useful for you.

There are interactions between the choice of sites and the choice of participants here. You may be concerned about the effects of your program on a particular population, which may be largely concentrated at one site.  In that case, if you have limited resources, you may want to evaluate only that site, or that site and one other.

Regardless of other considerations, you may want to set some guidelines about whom you include in the evaluation. How long do people have to be in the program, for instance, before they're included? In other words, what constitutes participation? (This also sets a criterion for who should be counted as a drop-out: anyone who starts, but leaves before meeting the standard for participation.) What about those whose attendance is spotty - a few days here, a few days there, sometimes with weeks in between? Do they have to have attended a certain number of hours to be considered participants?

These issues can be more complex than they seem. People may start and drop out of a program numerous times, and then finally come back and complete it. Many others start programs numerous times, and never complete them. It's usually impossible to tell the difference until someone actually gets to the point of completion, whatever that means for the particular program.

In a reversal of the start-many-times-before-completing scenario, there can be a few people who stay in a program right up till the end and then drop out. This may have to do with the fear of having to cope with success and a change in self-image, or it may simply be a pattern the person has learned to follow, and will have to unlearn before being able to complete the program.

Should any or all of these people be included in or excluded from an evaluation, either before (because of their history in the program) or after the fact? That's a decision you'll have to make, based on what their inclusion or exclusion will tell you. Just be sure that your evaluation clearly describes the criteria that you decide to use for your participants.

If you're an outside evaluator or academic or other independent researcher

Up to this point, we've largely ignored the evaluation difficulties faced by evaluators not directly connected with the organization or institution running the program they're evaluating. If you've been hired or designated by the organization or a funder to evaluate the program, you have to establish trust, both with the organization and its staff and with participants, if you hope to get accurate information to work with. You also have to learn enough in a short period about the community, the organization, the program, and the participants to devise a good evaluation plan, and to analyze the data you and others gather.

If you're an independent researcher - a graduate student, an academic, a journalist - you face even greater obstacles. First, you have to find a place to conduct your research - a program to evaluate - that fits in with your research interests. Then, you have to convince the organization running that program to allow you to do the research.Once you've jumped that hurdle, you're still faced with all the same tasks as an outside evaluator: establishing trust, understanding the context, etc.

Let's look first at the process you as an independent researcher might follow in order to choose and gain access to a setting appropriate to your interests. Once you've gained that access, you've become an outside evaluator, so from that point on, the course of preparing for the evaluation will be the same for both.

Choose a setting

If you're an academic or student, you can probably find an appropriate program by asking colleagues, professors, and other researchers at your institution. If none of them knows of one offhand, someone can almost undoubtedly put you in touch with human service agencies and others who will. Other possible sources of information include the Internet, funders, professional associations, health and human service coalitions, and community organizations. Public funding information is often available on the web, in libraries, or in newspaper archives. The wider you spread your net, the more likely you are to find the program you're looking for.

The right program will obviously vary depending on your research interests, but some questions that will inform your choice include:

• Does the setting include people who are actually experiencing the problem that is of interest to you?
• Is the setting similar to others of this type? (If not, its program might not be useful to others dealing with the issue, even if it works well in its own context.)
• Does the setting provide support for the research? Will staff, participants, and others help with data gathering, be forthcoming about context questions, cooperate with you?
• Does the setting have the resources to maintain the program after your evaluation is done?
• Does the setting permit the changes in operation required by the research?  If the planning of the evaluation and choosing of questions point to doing things differently, can and will the program make the necessary changes?
• Is the setting accessible?
• Accessibility includes not only handicap accessibility, but whether a site is in a neighborhood that feels welcoming or safe to participants, whether it is easily reachable by public transportation or on foot from the areas from which participants are drawn, and whether it is in a building or institution that doesn't feel intimidating or strange (a university campus or building can seem as threatening as a fortress to someone who is insecure about his educational background, for example.) Accessibility can be the determining factor in whether participants consider a program, or whether they stay in it.
• Is the setting stable? Are the program and organization stable enough that you know they'll be able to support their work at the current level, at least until the evaluation is completed?

Once you've found an appropriate setting, you'll have to convince the organization to collaborate with you on an evaluation. The next three steps are directed toward that goal.

Learn as much as you can about the organization you've chosen

Just as you wouldn't go to a job interview without doing some research about the employer, you shouldn't try to gain the cooperation of an organization without knowing something about it - its mission, its goals, whom it serves, who the director and board members are, etc. If someone told you about the organization, she may have, or may know someone who has, much of the information you need. If the organization maintains a website, much of that information will be available there. If it's incorporated, the office of the Secretary of the state of incorporation and/or other state offices will have information about the officers (i.e., the Board of Directors) and other aspects of the organization. Funding agencies may also have information that's a matter of public record, including proposals.

Contact the appropriate person(s) and request an interview

Find out whom (by name as well as position) you should talk to about conducting a research project in the organization you've chosen.

Depending on the organization, this could be the board president, the executive director, or the program director (if the program you're interested in is only part of a larger organization).  In any case, it might be wise to involve the program director even if he's not the final decision-maker, since his cooperation will be crucial for the completion of your research.

• If you can, get a personal introduction. It's always best if you come recommended by someone familiar with the person you need to speak with.
• If you can't get a personal introduction, it's usually best to send a letter requesting a meeting and explaining why, and follow it up with a phone call.
• Before the meeting, send a proposal outlining what you want to do. This should be substantive enough to help the organization decide whether it wants to work with you, but not so specific that it doesn't allow for collaborative planning of the evaluation.

Plan and prepare for the initial meeting

There are several purposes for this meeting, besides the ultimate one of getting permission and support for your project (or at least an agreement to continue to discuss the possibility). They include:

• Establishing your credentials - the experience, educational background, and any other factors that equip you to conduct this evaluation. This might include references from colleagues, professors, or other organizations you've worked with.
• Learning more about the program and the organization
• Explaining what you want to do and why, what form the evaluation results are likely to take, what you'll do with them, who'll have access, etc. This explanation should also cover issues of confidentiality and permission of participants.
• Explaining what you need from the organization and/or program - participation of participants and staff, for instance, any logistical support, access to records, or access to program activities
• Explaining what you're offering in return - your services for a comprehensive formal evaluation, any stipends, equipment or materials, other support services, or whatever else you may have to offer
• Clarifying the organization's needs, and discussing how they fit with your own - and how both can be satisfied

Assuming that your presentation has been convincing, and you're now the program evaluator, the rest of the steps here apply to both independent researchers and outside evaluators.

Find out all you can about the context

This may play out differently for outside evaluators than it does for independent researchers, but it's equally important for both. It means finding out all you can about the community, the organization, the program, and the participants beforehand - the social structure of the community and where participants fit in it, the history of the issue in question, how the organization is viewed, relationships among groups and individuals, community politics, etc.

If you're an outside evaluator, you can pick the brains of program administrators, staff, and participants about the community, the organization, and the issue. Ask them to steer you to others - community leaders, officials, longtime residents, clergy, trusted members of particular groups - who can give you their perspectives as well. If possible, get to know the community physically: walk and/or drive around it, visit businesses, parks, restaurants, the library. Understanding how the issue plays out in the community, the nature of relationships among groups and individuals, and what life is like in the neighborhoods where participants live will help a great deal in analyzing the evaluation of the program.

If you're an independent researcher, learn as much about the context as you can before you contact the program. Websites (for the organization and/or the community) and libraries are two possible sources of information, as are community and organization literature and people who know the community.  Learning about the community, the organization, and the participants beforehand will both help you determine whether this program fits with your research and help you advocate for its cooperation with your project. Once you have that cooperation, you can follow the same path as an outside evaluator (since that's what you are) to learn as much about the context of the program as you can.

Establish trust with program administrators, staff, and participants

This can be the most difficult part of an evaluation for someone from outside the organization. There's no magic bullet or predictable timeline, but there are several things you can do:

• Be yourself.  Don't feel you have to act a certain way: deal with people in the program as you do with friends and acquaintances in other circumstances. People can tell when you're being false, and are unlikely to trust you if you are.
• Treat everyone with equal respect, as colleagues in a research project.
• Don't assume you know more than anyone else just because you're the professional.
• Share freely what you do know, but don't tie yourself to any one process or method, especially in response to an opposite stance from a key individual.
• Ask administrators, staff, and participants what they want from the evaluation, and discuss how the evaluation could provide it.
• Don't be afraid to say "I don't know, but I'll find out," and then do.
• Follow through on whatever you say you'll do. Don't promise anything you can't deliver on, and make deadlines reasonable, so you can meet them.

General tips for all evaluators

These steps apply to everyone, internal evaluators as well as external.

Aim for a participatory evaluation

We've discussed above the involvement of all stakeholders to the extent possible. Involving participants, program staff, and other stakeholders in participatory planning and research can often get you the most accurate data, and may give you entry to people and places you normally might not have. On the other hand, participatory planning and research, as we've explained, takes time and energy. If you have limited time, you may not be able to set up a fully participatory project.  You can, however, still consult with stakeholders, and involve them in ways that don't necessarily involve training or large amounts of your time. They can help you line up interviews with participants or other important informants, for instance, and/or act as informants themselves about community conditions and relationships.

At least the people in charge of the program, and probably those implementing it as well, will expect to be part of the planning of the evaluation. They are, after all, the ones who need to know whether their work is effective, and how to improve it. Involving participants as well, in roles ranging from informants about context to actual researchers, is likely to enrich the quantity and quality of the information you can obtain.

Plan the evaluation, in collaboration with stakeholders

That collaboration should be at the highest level of participation possible, given the nature of the program, the time available, and the capacity of those involved (if program participants are five-year-olds, they probably have relatively little to contribute to evaluation planning...but their parents might want to be involved.)

The actual planning involves ten different areas, each of which will be the subject of one of the remaining sections in this chapter:

• Information gathering and synthesis
• Designing an observational system
• Developing and testing a prototype intervention
• Selecting an appropriate experimental design
• Collecting and analyzing data
• Gathering and interpreting ethnographic information
• Collecting and using archival data
• Encouraging participation throughout the research
• Refining the intervention based on the evaluation
• Preparing the evaluation results for dissemination

Once the planning is done, it's time to get started on conducting the evaluation. And when you're finished - having analyzed the information and planned and made the changes that were needed - it's time to start the process again, so that you can determine whether those changes had the effects you intended.  Evaluation, like so much of community work, is a process that goes on as long as the work itself does.  It's absolutely essential to the continued improvement of your program.

In Summary

Choosing evaluation questions - the areas in your work you'll examine as part of your evaluation of your program - is key to defining exactly what it is you're trying to accomplish. For that reason, those questions should be chosen carefully as part of the planning process for the program itself, so that the questions can guide your work as well as your evaluation of it. The more that stakeholders can be involved in that choice and planning, the more likely you are to create a program that successfully meets its goals serving the community.

Choosing those questions well entails understanding the context of the program - the community, participants, the culture of any groups involved, the history of the issue and of the social structure of the community and the organization - and (if you're an outside evaluator without ties to the program) establishing trust with administrators, staff members, and participants. That trust will enable you to conduct a participatory evaluation that draws on the knowledge and talents of all stakeholders, and to plan an evaluation that fits the goals of the program and accurately analyzes its strengths and weaknesses. With that analysis in hand, you'll be able to make changes to improve the program. Then you're ready to start the whole process again, so you can evaluate the effects of the changes you've made.

What do we mean by choosing questions?

___Evaluation questions are the questions your evaluation is meant to answer about your work

___Evaluation questions help set the direction of the work, as well as assess its effectiveness

___Ideally, choosing evaluation questions is part of the planning of the overall program

Questions to ask yourself as you choose evaluation questions:

___What do you want to know?

___Why are you interested?

___Is the issue you’re addressing important to the community or to the society? 

___How does the issue relate to the field?

___Is the issue general, rather than specific to your population or community?

___Who might use the results of your evaluation? 

___Whose issue is it?

Why is it necessary to choose evaluation questions carefully?

___It helps you understand what effects different parts of your effort are having

___It makes you clearly define what it is you’re trying to do

___It shows you where you need to make changes

___It highlights unintended consequences

___It guides your future choices

___In participant evaluations, it involves stakeholders in setting the course of the program, thus making it more likely that it meets community needs

___It provides focus for the evaluation and the program

___It determines what needs to be recorded in order to gather data for evaluation

When should you choose questions and plan the evaluation?

___If possible, choosing questions and planning the evaluation should be an integral part of planning your program

___If your reality makes that impossible, choosing questions and planning the evaluation should take place as soon as possible after the program starts

Who should be involved in the process?

___To the extent possible, the process should involve all stakeholders, including program participants and beneficiaries

How do you choose questions?

___Describe the issue or problem you’re addressing

___Describe the importance of the problem

___Describe those who contribute to the problem

___Assess the importance and feasibility of changing those behaviors

___Describe the change objective

___Make sure that the expected changes would constitute a solution or substantial contribution to the problem

How do you plan the evaluation?

___Take into account the issues raised by multiple or very different settings

___Take into account the issues raised by participant groups that differ in culture, ability to complete the program, geographical location, and other factors

For outside evaluators, specifically:

___Choose a setting

___Learn as much as you can about the organization you’ve chosen

___Contact the appropriate person(s) and request an interview

___Plan and prepare for the initial meeting

For all evaluators:

___Find out all you can about the context

___Establish trust with program administrators, staff, and participants

___Aim for a participatory evaluation

___Plan the evaluation in collaboration with stakeholders

Consider all the elements of an evaluation in your planning:

___Information gathering and synthesis

___Designing an observational system

___Developing and testing a prototype intervention

___Selecting an appropriate experimental design

___Collecting and analyzing data

___Gathering and interpreting ethnographic information

___Collecting and using archival data

___Encouraging participation throughout the research

___Refining the intervention based on the evaluation

___Preparing the evaluation results for dissemination

Example 1: Using Miro Board for Collaborative Development of Evaluation Goals and Questions

 

Noah Goodman shares the process he used to prepare for and conduct a kick-off meeting to develop evaluation goals and questions for the Right Question Institute.

Integrating Technology in Evaluation TIG Week: Using Miro Boards to Engage Clients in Collaborative Development of Evaluation Goals and Questions by Noah Goodman

Suppose you wanted to design a house that used very little energy, required relatively few resources to build and maintain, and was affordable for many families. You might have some original ideas about how this could be done, but you would also want to learn from what others have tried. You might read about earth-bermed houses (houses built partly into a hillside or earth mound), solar panels or small wind turbines for producing electricity, energy-efficient windows and insulation, wastewater recycling, and lower-toxicity building materials that incorporate reclaimed wood or metal. You might talk with people who have built or lived in energy-efficient homes to learn about their experiences. You would also want to understand barriers to environmentally sustainable strategies and approaches communities have used to address those barriers. There is a great deal of information available, and gathering relevant evidence and experience can help you identify useful approaches, adapt them to your circumstances, and develop new ideas based on what has already been learned.

The same is true if you are designing an intervention or program to address a community health or other issue, or an evaluation of that program. Other communities and organizations have likely addressed similar issues, with varying results. Understanding what they did, how they did it, the context in which they worked, and what resulted can help you make informed decisions about your own effort. You may find methods or strategies from different sources that can be adapted and combined to fit the priorities, strengths, and conditions of your community. You may also learn that an approach you were considering has repeatedly produced limited or unintended results and should be reconsidered.

Gathering and using others' ideas does not mean giving up your own ideas or the opportunity to innovate. New approaches often develop by building on previous knowledge, experience, and experimentation. Understanding what has already been tried can help you identify opportunities to adapt, improve, or develop new approaches.

This section focuses on gathering information about a community issue and previous efforts to address it, then bringing that information together to inform program and evaluation design. Although this chapter focuses on evaluation, much of the material in this section also applies to planning the program or intervention itself. Program design and evaluation are closely connected.

An evaluation is a form of systematic inquiry: we are trying to understand what works, for whom, under what circumstances, and why. Many of the steps involved in designing an evaluation are also relevant to designing the program being evaluated. Evidence from previous efforts, community knowledge, and new ideas can help shape both an intervention and the questions used to evaluate it. Although this section discusses program design, the same principles apply to evaluation design.

What do we mean by information gathering and synthesis?

Information gathering refers to collecting relevant information about the issue you are addressing and about how other organizations, communities, researchers, and practitioners have responded to it. The more you understand about the issue, its context, and previous approaches, the better prepared you will be to design an effective and appropriate program or intervention.

Information can come from many sources, depending on what you need to learn. In general, you can consult existing published or documented sources and examine real-world examples of programs and interventions that have addressed similar issues. We will introduce both types here and discuss them in more detail later in the section.

• Existing sources. These include published and documented materials that can help you understand an issue or previous attempts to address it. Sources may include scholarly research, professional publications, government reports, community reports, journalism, organizational materials, and statistical or demographic data. Different sources provide different forms of evidence and perspective, so it is useful to consider their purpose, quality, methods, limitations, and relevance to your context.
• Real-world examples. These are programs, policies, practices, or interventions that communities and organizations have actually implemented. Studying them can help you understand what worked, what did not, under what conditions, and why. Information may come from people directly involved in these efforts, program documentation, evaluations, case studies, community reports, or research literature in fields such as community psychology and public health.

Synthesis means bringing together information from multiple sources to form a coherent understanding or approach.

In this section, synthesis refers to analyzing what you have learned through information gathering and combining relevant evidence, community knowledge, practical experience, and ideas into an approach that fits the priorities and context of the community you are working with.

Synthesis requires identifying the functional elements of the programs, policies, or ideas you examine. Functional elements are the core components—such as methods, frameworks, activities, strategies, relationships, policies, and implementation practices—that contribute to how an approach works. Once you understand these elements, you can determine which may be relevant to your community, which may need adaptation, and how they might fit with community knowledge and locally developed ideas.

Information gathering and synthesis are important both when developing a program and when designing an evaluation that can help strengthen it. Both begin with understanding the issue, identifying the people and communities involved, considering the conditions that influence it, and developing an approach for creating change. These decisions help define short- and long-term goals and, in turn, shape the evaluation questions. A meaningful evaluation depends on clearly understanding what the program is intended to accomplish and how it is expected to contribute to change.

Why gather and synthesize information?

If you are developing a program to address a community issue, such as violence prevention or early childhood education, you may already have substantial knowledge and experience related to that issue. Community members, practitioners, and other stakeholders may also have strong ideas about what could work. Why take additional time to review research, examine other programs, and speak with people involved in similar efforts? Because gathering and synthesizing information can help you build on existing knowledge, avoid preventable problems, understand your local context more fully, and strengthen the overall design of your program.

• It can help you avoid repeating avoidable mistakes. Other organizations and communities have likely addressed similar issues. Some efforts may have been effective, some may have produced mixed results, and some may not have worked as intended. Each can provide useful lessons. Understanding previous efforts can help you build on promising approaches while avoiding strategies that have repeatedly created problems or failed to achieve their goals.

Having an idea similar to one that has already been tried is not a problem. In fact, learning that an approach has worked elsewhere may strengthen the case for considering it. At the same time, discovering that a similar approach has repeatedly struggled can help you examine why and decide whether it should be adapted, replaced, or avoided. Learning from previous experience can save time and resources while improving the likelihood that your program will be useful and sustainable.

• It can help you develop a deeper understanding of the issue. Effective action starts with understanding what you are addressing. This may include immediate and underlying causes, historical and structural conditions, community strengths and resources, how people experience the issue, its consequences, and who or what has the ability to influence change.
• It gives you more tools for designing an effective program. Information is one of the most important resources for planning and implementation. Understanding what has worked, for whom, under what conditions, and why can help you select strategies that address multiple dimensions of an issue rather than relying on assumptions or a single approach.
• Most approaches are not one-size-fits-all. The more information you gather, the more approaches, methods, and frameworks you can consider. The goal is not simply to copy another program, but to identify elements that can be appropriately adapted to your community's context, priorities, strengths, and needs.
• It can help you design culturally responsive approaches. Information gathering can help you better understand the histories, cultures, values, languages, strengths, and priorities of the communities involved. It can also help identify approaches that have been used in similar contexts. This information should complement—not replace—direct engagement with community members and their knowledge about what is appropriate and meaningful in their own community.
• Learning from different settings and perspectives can generate new insights and ideas. New approaches often emerge from combining existing knowledge in new ways. Looking across fields, communities, cultures, and geographic settings can expose you to different ways of understanding an issue and expand the range of possible strategies. Those ideas can then be examined and adapted with community members to determine whether they are relevant locally.

When should you gather and synthesize information?

Information gathering and synthesis are important to both program effectiveness and meaningful evaluation. They should begin early in the planning process and continue throughout the life of the program. Ongoing learning can help you identify emerging evidence, changing community conditions, new opportunities, and possible improvements.

Major changes may be easier to assess when they occur after an evaluation cycle, when you have enough information to understand what was working and what was not. At the same time, smaller adjustments may be necessary during implementation to respond to participant feedback, accessibility concerns, emerging risks, or changing circumstances.

There can be tension between continuously adapting a program and determining whether a particular approach is effective. If a method or activity changes substantially during an evaluation period, it may become more difficult to determine which version of the approach contributed to the results.

How much adaptation is appropriate depends partly on the purpose of the program and evaluation. If the primary purpose is to test whether a specific approach works as designed, maintaining consistency may be particularly important. If the primary responsibility is to respond to the needs and well-being of current participants and communities, changes may need to be made as new information becomes available. When changes are made, documenting what changed and why can help evaluators interpret the results.

There can also be ethical issues involved. In research involving new medical treatments, for example, participants may be assigned to different study groups under an approved research protocol and informed-consent process. If evidence shows that an intervention is causing harm, researchers have an obligation to respond appropriately. Likewise, strong evidence that one approach provides substantial benefit may raise ethical questions about continuing to withhold it from other participants. Community programs may raise different ethical considerations, but potential benefits, harms, fairness, informed participation, and participant well-being should always be considered.

Who should gather and synthesize information?

Throughout this chapter, we assume that planning, design, implementation, and evaluation involve multiple stakeholders and forms of knowledge.

Stakeholders in a community program or intervention might include:

• Program participants and community members directly affected by the issue
• Program staff, volunteers, and administrators
• Others whose work or responsibilities intersect with the issue, such as health professionals, educators, public safety personnel, social service providers, or community-based organizations
• Researchers and subject-matter experts
• Local officials and policymakers
• Community advocates, organizers, and leaders

Information gathering

In a participatory process, information gathering can be strengthened by sharing responsibilities according to participants' knowledge, interests, skills, relationships, and opportunities to contribute. Researchers or others with experience reviewing academic literature may take the lead on some evidence reviews. Community members may bring important knowledge about the history of the issue, how it is currently experienced, and which sources or perspectives are most relevant locally. Program staff may have connections with organizations doing similar work. Participants with experience using databases, online research tools, policy analysis, or other methods may support those areas of the work. Roles should be determined collaboratively rather than assigned according to assumptions about people's backgrounds.

Training and support can also expand who is able to participate in different parts of the research process. People who are new to research can learn to interpret demographic information, review program materials, conduct interviews, contact organizations, analyze policy, or use research tools. The goal should be to identify what support, technology, accessibility accommodations, language access, or training participants need to contribute in ways that are meaningful to them.

Synthesis

It is especially important for people with different perspectives to participate in interpreting and synthesizing the information. Participants who bring lived experience or deep community knowledge may recognize patterns, assumptions, or implications that are less visible to professional researchers. Researchers and practitioners may contribute methodological or subject-matter expertise. Bringing these perspectives together can improve both interpretation and decision-making.

Information gathering and synthesis, like the rest of a participatory process, should reflect the priorities, knowledge, interests, strengths, and accessibility needs of stakeholders.

How do you gather and synthesize information?

Gathering and synthesizing information involves several steps. Many can be carried out collaboratively as part of a participatory process, while particular research tasks may be shared among individuals or smaller groups.

Decide what you need to know

The first step in gathering information is determining what you need to understand. Areas to explore may include:

• Details about the issue. These may include immediate and underlying causes; historical, social, economic, organizational, environmental, or policy conditions; effects on individuals and communities; how the issue has changed over time; community strengths and protective factors; and previous efforts to address it.
• How the issue has been addressed elsewhere. This might include evidence-informed or evidence-based practices, promising approaches, community-developed strategies, adaptations used in different settings, and approaches that produced limited or unintended results. Learning why an effort was unsuccessful can be just as valuable as learning why another succeeded.
• People and organizations who can contribute knowledge or support. This may include people with lived experience, community organizations, practitioners, researchers, evaluators, policymakers, advocates, and people involved in previous efforts to address the issue.
• Who is affected locally, and how. Consider which communities or groups experience the issue most directly and whether impacts differ by geography, race, ethnicity, culture, income, disability, age, gender, or other relevant characteristics. Also consider people who may be affected less visibly, including family members, caregivers, service providers, educators, employers, or others connected to those directly affected.
• The importance of the issue to the community. This includes at least two questions:
  • How important do community members consider the issue to be?
  • How and to what extent does the issue affect the community as a whole?
• Community priorities, strengths, and needs related to the issue. What resources, supports, opportunities, relationships, services, policies, or environmental changes could improve the situation? What existing community strengths can be built upon? Which approaches do community members consider acceptable, relevant, and feasible?
• Other contextual information. This may include community history, relationships among groups and organizations, prior experiences with institutions or programs, culture, language, economic conditions, transportation, geography, policy, and other factors relevant to the work.
• Who or what can influence change. Depending on the issue, this may include public officials, policymakers, businesses, property owners, government agencies, schools, employers, health systems, community organizations, funders, service providers, and people directly affected by the issue. It may also include policies, organizational practices, resource allocation, institutional systems, and environmental conditions.

Determine your likely information sources

As discussed above, information can come from published or documented sources as well as real-world experience. Published sources may include scholarly research, professional publications, government and community reports, journalism, organizational documents, and statistical information. Depending on your questions, you may use several types of sources to understand both the evidence and the community context.

The Internet provides access to a vast range of information, including academic research, government data, laws and regulations, program evaluations, organizational reports, community resources, statistical databases, professional guidance, and descriptions of promising practices. Libraries, government databases, university repositories, research organizations, and organizational websites can all be useful starting points.

Online information varies widely in quality. Consider who produced the information, what evidence it uses, when it was published or updated, what methods were used, whether potential conflicts of interest are disclosed, and whether other credible sources support its claims. General-reference websites, social media, discussion forums, and other informal sources can help identify questions, terminology, or additional sources, but important decisions should be supported by information whose credibility and relevance you can evaluate.

Existing sources

Scholarly and professional sources might include:

• Peer-reviewed academic and professional journals
• Academic and professional books
• Doctoral dissertations and master's theses available through universities, libraries, and research repositories
• Papers, posters, reports, and proceedings from academic and professional conferences
• Evidence reviews, professional publications, and reputable science and research journalism
• Newspaper and journalism archives
• Direct communication with researchers, evaluators, practitioners, and others with relevant expertise
• Professional networks, research communities, mailing lists, webinars, and other field-specific forums

Public-facing sources of information might include:

• Books written for general audiences
• Articles in reputable magazines, news outlets, and science or health publications
• Investigative reporting, explanatory journalism, podcasts, documentaries, and other media that provide useful context or identify additional sources

Statistical and demographic information may be available from:

• Census and other government data sources
• Community reports, community health assessments, community report cards, strategic plans, and needs or strengths assessments
• Organizational, agency, school, health system, and administrative data, when those data are available and appropriate to use

News and other media can also call attention to community issues, emerging research, or promising programs. Media coverage usually provides only part of the picture, so it is often most useful as a starting point for identifying additional sources, organizations, researchers, or questions to investigate.

Real-world examples

Potential sources of information about real-world programs and approaches include:

• Program directors, coordinators, and staff
• Colleagues and professional contacts
• Funders and public agencies
• Leaders and members of community coalitions, partnerships, and networks
• Officials or organizations coordinating community-wide efforts
• Community members directly affected by the issue
• Current or former program participants
• People whose work intersects with the program, such as educators, health professionals, social service providers, public safety personnel, or other community partners
• Community members with relevant knowledge, experience, or relationships
• Researchers, practitioners, evaluators, and other people with relevant expertise
• Your own experience and observations in the community

Look broadly when searching for useful models or new ideas. Approaches developed in another field, community, or geographic region may offer concepts that can be adapted to your work. Public health, education, social services, community development, urban planning, behavioral health, environmental health, and other fields often address overlapping social and community conditions. The important question is not whether an approach comes from the same field, but whether its underlying principles and components are relevant and adaptable to your context.

Devise a plan for collecting information

Consider the following questions:

• Who will gather what information? A strong information-gathering team includes different forms of knowledge, experience, relationships, and skills. Responsibilities can be shared according to participants' interests and strengths while also creating opportunities for people to learn new skills. Researchers may contribute experience reviewing academic literature, for example, while community members may have knowledge and relationships that are essential for understanding local history, priorities, and experience. These roles should remain flexible and should not be based on assumptions about who is capable of doing particular kinds of research.
• How will the information be gathered? Written and online sources may be reviewed through libraries, databases, organizational records, and websites. Information from people may be gathered through formal or informal interviews, observation, surveys, public meetings or community conversations, focus groups, participatory methods, or other approaches. Methods should reflect the questions being asked, the time and resources available, participant preferences, accessibility, privacy, and the depth of information needed.
• What support will participants need? People may have different levels of experience with research, technology, academic writing, data interpretation, or particular languages. The group may need interpretation or translation, accessible formats, assistive technology, transportation, childcare, technology access, research training, or support interpreting specialized materials. Identify these needs collaboratively and create ways for everyone to participate meaningfully.
• What is the timeline for information gathering? Learning should continue throughout the life of a project, but the initial information-gathering phase usually needs a clear timeframe so that planning does not indefinitely delay action. The timeline should reflect the urgency and complexity of the issue, community priorities, available resources, external deadlines, and the amount of information needed to make informed decisions.

Collect information

Once the plan is established, begin gathering information. Provide any training, tools, accessibility supports, or resources participants need, and make sure responsibilities are clear. Regular check-ins during the process can help participants share what they are learning, identify gaps, support one another, adjust the research plan, and begin interpreting emerging findings. These conversations can also make synthesis more manageable by allowing the group to process information throughout the research process rather than all at once.

Synthesize: Take it all apart

Synthesis begins by breaking information into useful components, examining how those pieces relate to one another, and identifying which ideas, strategies, and conditions are most relevant to your community.

Three major areas are usually important to consider:

• What is known about the issue itself. What individual, interpersonal, organizational, community, environmental, policy, and structural factors influence the issue? What are its immediate and underlying causes? What strengths or protective factors already exist? Which factors are within the scope of your effort, and which require broader partnerships or systems-level change? What are the potential consequences for individuals and the community if the issue is not addressed?
• The community context of the issue. Who is affected, and in what ways? Are some groups affected differently or disproportionately? How does the issue affect families, relationships, organizations, neighborhoods, and the wider community? What has the community's experience with the issue been? What previous efforts have been attempted? What local strengths and resources can support change? Are there policies, systems, practices, relationships, or environmental conditions that also need to change?
• Previous efforts to address the issue. Information may come from research literature, evaluations, community knowledge, case studies, program documentation, and people directly involved in those efforts. Examine the specific components of each approach. What strategies were used? What training or support was provided? What information and resources were available to participants? What organizational, environmental, or policy changes were involved? What assumptions or principles shaped the approach? Which aspects of the issue did it address? In what kinds of communities and contexts was it used? Who participated? Who benefited, and who may have been left out? What were the short- and long-term outcomes? What conditions contributed to success, limited effectiveness, or unintended consequences?

An unsuccessful program does not necessarily mean that every element of the program was ineffective. Some components may have been useful while others created barriers or were based on incorrect assumptions. A promising strategy may also fail because it was poorly implemented, lacked resources, was not culturally or contextually appropriate, excluded important stakeholders, or was introduced under conditions that made success unlikely. Understanding why an effort produced its results is often more useful than simply labeling it successful or unsuccessful.

It is also important to distinguish between identifying an approach that appears to work and understanding the conditions under which it works. Leadership, staffing, relationships, funding, community trust, organizational capacity, implementation quality, policy context, and other factors can strongly influence outcomes.

When examining programs for ideas, look beyond the program name or overall model. Identify the specific components, relationships, implementation practices, and contextual conditions that appear to contribute to its results.

Synthesize: Put it back together

Analyze what you have learned to determine which ideas and components are appropriate for your community and how they might fit together.

• What approaches have been used with communities or groups whose circumstances are similar to yours? Were successful approaches highly specific to their original setting, or do their underlying principles appear adaptable?
• Which components could be adapted for your context while preserving the elements that make them effective?
• What is missing from the approaches you reviewed? Are important community priorities, strengths, populations, barriers, or structural conditions not being addressed?
• Does the information confirm, challenge, or complicate what you previously believed about the issue?
• What features of your community or situation differ from the settings in which other approaches were developed? How should those differences affect adaptation and implementation?
• What does the information suggest about addressing underlying conditions such as economic inequity, social exclusion, unequal access to resources, or limited decision-making power?
• At what levels did successful efforts focus their strategies—individual, interpersonal, organizational, community, policy, or systems? Which combination is most relevant to your context?
• Perhaps most importantly, how does your community define success? Which approaches came closest to producing the outcomes that stakeholders value, and which components appear most relevant to the priorities and conditions you identified?

Answering these questions can help you identify components that may be appropriate for your community, determine what should be adapted, generate new ideas, and recognize approaches that may not fit your goals or context.

Synthesis is not simply a matter of combining unrelated program components. Begin with a clear framework informed by your vision, mission, community priorities, evidence, and understanding of how change is expected to occur. The components you select should reinforce one another and contribute to a coherent strategy. When elements conflict or are combined without understanding how they function, the resulting program may be difficult to implement or evaluate effectively.

Keep at it

Information gathering and knowledge synthesis should continue throughout the program. New evidence may emerge, community conditions may change, participants may identify new concerns or strengths, and additional approaches may become available. Even when major program changes are postponed until an evaluation cycle is complete, continuing to learn helps prepare the program for thoughtful adaptation. Ongoing reflection and learning can help keep the work responsive, relevant, and effective.

In Summary

Gathering existing information about a community issue and previous efforts to address it is an important part of planning both programs and evaluations. Bringing together research evidence, community knowledge, lived experience, lessons from previous efforts, and local context can help you design an approach that builds on what has been learned while creating room for local adaptation and innovation.

Information gathering and synthesis should continue throughout the life of the program. Ongoing learning can help communities and organizations understand changing conditions, strengthen promising approaches, recognize ineffective or harmful strategies, and adapt the work in ways that remain responsive to community priorities and evidence.

What do we mean by information gathering and synthesis?

___Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it

___You can gather information using both existing sources and natural examples

­­___Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something that meets the needs of the community and population you’re working with

___Synthesis involves extracting the functional elements of both the analysis of the issue and approaches to it

___Functional elements are those that are indispensable either to understanding the issue, or to implementing a particular program

Why gather and synthesize information?

___It will help you avoid reinventing the wheel

___It will help you to gain a deep understanding of the issue so that you can address it properly

___You need all the tools possible to create the best program you can

___It’s likely that most solutions aren’t one size fits all

___It can help ensure your program is culturally sensitive

___Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program

When should you gather and synthesize information?

___Information gathering and synthesis should continue throughout the life of the program

Who should gather and synthesize information?

___Information gathering and synthesis is often most effectively conducted by a multi-sectoral participatory group including all stakeholders in the issue

How do you gather and synthesize information?

___Decide what you need to know about the issue itself, successful and unsuccessful attempts to address it in various circumstances, and the local context

___Determine your likely sources for the various types of information you’re seeking

• Existing sources include scholarly, mass-market, and statistical/demographic published information
• Natural sources include some published information about programs, but can best be obtained by direct contact with those involved in planning, implementing, or participating in programs relevant to your issue
• It’s important to pay attention to both successful and unsuccessful attempts to address the issue, and to step outside your own field in search of solutions that work

___Devise a plan for gathering information

• Decide who will gather what information
• Decide how information will be gathered
• Decide what adjustments will be made for gaps in experience or skills
• Set a timeline for the initial information gathering

___Collect information

___Begin synthesis by taking it all apart – extract the functional elements of what you’ve learned

___Complete synthesis by putting the relevant pieces back together as a coherent program that speaks to your community’s needs

___Keep at it by continuing to gather and synthesize information throughout the life of the program

• What do we mean by an observational system?

• Why design an observational system?

• When should you design an observational system?

• Who should design an observational system?

• How do you design an observational system?

A local community health center was starting a program to support regular physical activity among people with high blood pressure. The program had one main objective: to help participants engage in about 45 minutes of moderate aerobic activity at least four times a week over the course of six months. The hope was that regular physical activity would help participants manage their blood pressure, improve cardiovascular health, and support their overall well-being. A related goal was for participants to identify forms of physical activity they enjoyed and could continue after the formal program ended.

The center recruited 50 people who wanted to participate. After appropriate health assessments, participants attended workshops about nutrition, high blood pressure, physical activity, injury prevention, and ways to incorporate movement into daily life. They also discussed different forms of activity—such as walking, bicycling, swimming, or other activities appropriate to their interests and abilities. Rather than requiring everyone to use the same exercise routine, the center encouraged participants to choose activities that worked for them while working toward the overall activity goal. Participants were asked to keep activity logs and to meet in small groups with health center staff once a month to review blood pressure, discuss progress and barriers, and receive support and guidance.

The center then had to decide how to evaluate the program. One important question was whether participants were able to maintain their physical activity routines over the six-month period. Because participants exercised independently and at different times and places, staff could not directly observe every activity session. The center also wanted to understand whether participants experienced changes in blood pressure, well-being, physical activity habits, and other relevant health indicators. How could the center design a system to gather accurate, respectful, and useful information about what participants were doing and what changes were occurring?

Once you have identified your evaluation questions and determined what information you need, you need a systematic way to collect that information. That is the purpose of an observational system. Like the center in the example, you may need several ways to examine behaviors, experiences, conditions, program activities, and changes over time. This section focuses on designing observational systems that provide useful information while respecting participants' privacy, dignity, and rights.

What do we mean by designing an observational system?

An observational system is a structured way of gathering information about a program—what is being implemented, how people are participating, what conditions are present, and what changes appear to be occurring. "Observation" may involve directly watching activities or conditions, but it can also include other systematic ways of documenting program processes and outcomes. The appropriate methods depend on the evaluation questions, the setting, the people involved, and ethical and privacy considerations.

• Direct observation. Direct observation involves systematically observing activities, behaviors, settings, or conditions firsthand. For example, if an initiative aims to increase use of a public park, observers might document how the park is used at different times of day, during different weather conditions, or during special events. Observation in public settings should still be planned carefully, with attention to privacy, research ethics, and whether consent or public notice is appropriate. In program settings, participants should generally understand when and why observation is taking place.
• Participant observation. A participant observer takes part in the setting or activity while also documenting what they experience and observe. In the park example, a neighborhood resident involved in the initiative might document patterns of park use while participating in everyday activities there. Participant observation can provide important contextual information because the observer experiences the setting from within it, but observers should be transparent about their role when appropriate and follow relevant ethical and privacy guidelines.
• Self-reports. Some experiences and behaviors cannot or should not be directly observed. People may be asked to describe their own experiences through interviews, journals, surveys, activity logs, questionnaires, or other forms of first-person reporting. Self-reports can provide information that is unavailable through external observation, including people's motivations, experiences, perceptions, and activities outside a program setting. Like all methods, self-reporting has limitations, so it may be useful to combine it with other appropriate sources of information when possible.
• Reports from others. An observational system may also include information from people who have relevant knowledge of the conditions or activities being evaluated. Depending on the program, these might include educators, health professionals, family members, community partners, service providers, or other people with direct and appropriate knowledge. Information should be collected in ways that protect confidentiality and respect participants' privacy and consent.
• Electronic or technology-assisted observation. Devices such as activity trackers, environmental sensors, cameras, audio recorders, or health-monitoring equipment can sometimes provide useful information. These tools should be used only when appropriate to the evaluation and with careful attention to informed consent, privacy, data security, accessibility, and how collected information will be stored and used. The fact that a technology can collect information does not necessarily mean that collecting it is appropriate.
• Tests and assessments. Depending on what is being evaluated, assessments may include academic measures, skills demonstrations, health screenings, validated questionnaires, or other tools. Assessments should be appropriate for the population and purpose, and their limitations should be considered when interpreting results.
• Public and administrative records. Census information, public health data, employment statistics, school or organizational records, and other administrative data may provide information about community-level conditions or outcomes. Evaluators should consider data quality, privacy requirements, who is represented or missing from the data, and whether the information is appropriate for the questions being asked.
• Products or results of behavior. Sometimes it is more practical or appropriate to examine the results of an activity rather than observing the activity itself. For example, an environmental initiative might measure changes in water quality, litter, or air pollution rather than attempting to observe every behavior that contributes to those conditions. A school-based health initiative might examine participation, student-reported well-being, access to healthy meals or physical activity, or other appropriate indicators rather than relying on body size as a primary measure of success.

In addition to deciding which methods to use, an observational system should specify when, where, how often, by whom, and under what circumstances information will be collected. These decisions should be guided by the evaluation questions and by ethical, practical, cultural, and accessibility considerations.

Consider whether you want to examine the process of the effort—how the program was planned and organized and whether implementation matched the intended approach. You may also want to examine what was actually delivered, who participated, how long activities lasted, what adaptations were made, and which parts participants found useful. Finally, consider which outcomes or changes you want to understand and how those outcomes can be measured responsibly.

Designing an observational system requires thinking carefully about what you need to know and selecting methods that can provide that information accurately, consistently, ethically, and feasibly. We will discuss this process in more detail below.

Why design an observational system?

If you're serious about evaluation, there are several reasons to design a strong observational system:

• It can help you collect reliable information. A system that clearly defines methods, timing, procedures, and responsibilities makes information collected across observers, settings, and time periods more consistent and useful.
• It can help you collect the information you actually need. A well-designed system focuses data collection on the evaluation questions rather than gathering large amounts of information without a clear purpose.
• It can make consistent data collection more likely. When the people collecting information understand and agree on the procedures, observations are more likely to happen when and how they are intended.
• It can make analysis easier. Consistent observations can support both quantitative analysis, which examines numerical patterns, and qualitative analysis, which examines experiences, meaning, context, and interpretation.
• It can help you avoid fragmented evaluation. A structured system helps connect the information you collect directly to the questions you are trying to answer.
• It can strengthen confidence in your findings. Clearly documented and consistently applied methods make it easier to explain how conclusions were reached and to identify the limitations of the evidence.
• It can strengthen accountability with communities, funders, and policymakers. A thoughtful evaluation based on systematically collected information can help stakeholders understand what the program accomplished, what remains uncertain, and what should happen next.
• It can help you share promising practices responsibly. Strong evidence can help you explain which program components appear effective, for whom, and under what conditions, rather than assuming that an approach that worked in one setting will automatically work everywhere.
• It can provide useful information about what is working, what is not, and what should be adjusted.

When should you design an observational system?

An observational system includes the methods used to examine and document program processes, activities, experiences, and outcomes. Because it is central to evaluation, the ideal time to design the system is before implementation begins. Doing so allows you to establish baseline information and monitor the program from the start.

In practice, many community organizations begin formal evaluation after a program is already underway because of limited time, staffing, funding, or evaluation capacity. Whenever evaluation begins, the observational system should be designed around the questions you are trying to answer and the information that can still be gathered accurately. It is worth taking the time to develop a system that is realistic, ethical, and useful.

Whenever possible, observations should cover a full program cycle from beginning to end. Some programs do not operate in fixed cycles, in which case observation may focus on individual participants, particular activities, implementation periods, or changing community conditions.

Records created before a formal evaluation begins—such as program notes, activity logs, meeting records, or participant feedback—may provide useful information. Before including them, determine whether they were collected consistently enough and for purposes compatible with the current evaluation. Information created for one purpose should not automatically be treated as equivalent to data collected under a defined observational protocol.

Beginning observation late may also mean that important early changes or implementation decisions have already occurred. When feasible, begin collecting relevant information early enough to establish a baseline and understand how the program develops over time.

Who should design an observational system?

The Community Tool Box emphasizes participatory research and evaluation. An observational system is generally stronger when the people who will collect information, the people whose experiences are being represented, and those who will use the findings have meaningful opportunities to contribute to its design.

In smaller community-based organizations, staff and volunteers may already have multiple responsibilities. The observational system should therefore be realistic enough to carry out with available time, staffing, technology, and other resources. Involving the people responsible for implementation can help identify unnecessary burdens and make the system more practical.

The design process should include people who will actually collect the information, along with people who understand the program and the communities involved. Researchers or evaluators can contribute methodological expertise, while program participants and community members can help ensure that the system reflects community priorities, cultural context, accessibility needs, privacy concerns, and the realities of participation. The design team may therefore include members of the broader evaluation group as well as people specifically recruited to help develop the observational system.

The group might include:

• Program staff, volunteers, and administrators
• Staff responsible for records, data management, technology, or administrative support
• External evaluators, researchers, or research consultants
• Program participants and community members affected by the work
• Community partners or trained volunteer observers

If the design group does not include anyone with evaluation or research experience, participants may benefit from training or technical assistance on observational methods, ethics, privacy, measurement, and the kinds of information different methods can provide. If experienced researchers are involved, they can share this knowledge as part of a collaborative design process rather than assuming that methodological expertise should determine the system without community input.

How do you design an observational system?

Once you have developed your evaluation questions and overall evaluation plan, the next step is deciding how you will gather the information needed to answer those questions.

Review your evaluation questions

Return to what you originally wanted to understand about the program. In the health center example, the program was designed to support regular physical activity among people with high blood pressure, with participants working toward approximately 45 minutes of moderate activity four times a week over six months.

The program hoped to contribute to several outcomes:

• Participants would experience improved blood pressure management
• Participants would increase or maintain regular physical activity in ways appropriate to their health, abilities, and preferences
• Participants would report improvements in overall well-being
• Participants would continue forms of physical activity they found sustainable after the six-month program ended

Some evaluation questions might therefore include:

• To what extent were participants able to engage in their planned physical activity during the program?
• How did participants' blood pressure change over the six-month period?
• What changes did participants report in physical functioning, well-being, or confidence in maintaining regular activity?

Other questions might include:

• How well attended were the workshops? Did participants find them useful, accessible, and relevant? Were particular program components associated with stronger engagement or outcomes?
• Did participants report changes in their overall well-being by the end of six months?
• Did participants continue engaging in regular physical activity after the formal program ended, and were improvements in blood pressure or well-being sustained?

These may represent only some of the information the center wants. Your own evaluation may also examine planning, implementation, participant experience, accessibility, equity, timelines, adaptations, and intermediate benchmarks. Each of these questions should inform the design of the observational system.

Decide what you need to observe to answer your questions

The kinds of information you need will depend on the program, community context, and evaluation questions.

Common areas include:

• Participants' behavior or activities. This might include participation in physical activity, demonstrated skills in an employment-training program, use of community resources, or interactions in a program setting. Behaviors should be defined carefully and observed in ways that respect participants' privacy and dignity.
• The behavior or practices of others involved. An evaluation might examine how staff interact with participants, how institutions implement a policy, or how other groups respond to an intervention. Examining organizational and staff practices can be particularly important when program outcomes depend on more than participant behavior alone.
• Conditions. An initiative may aim to change physical, social, environmental, organizational, or policy conditions—for example, improving affordable housing, restoring a polluted waterway, increasing accessibility, or changing institutional practices.
• Products or results of behavior or conditions. When an activity cannot or should not be observed directly, you may need to measure relevant outcomes or indicators. For example, a sexual health program might examine rates of sexually transmitted infections, access to preventive services, contraceptive use reported confidentially by participants, or other appropriate indicators rather than attempting to observe private behavior directly.

When you rely on indirect indicators, make sure they are meaningfully connected to the behavior or condition you are interested in. Consider other factors that could influence the same outcome, and avoid treating one indicator as definitive evidence when multiple explanations are possible.

• Participants' knowledge, perceptions, or attitudes. These might be assessed through interviews, surveys, validated scales, knowledge assessments, or other appropriate methods.
• The knowledge, perceptions, or attitudes of other groups. An advocacy initiative, for example, might examine changes in public understanding, decision-maker awareness, organizational practices, or community support for a policy.
• Goal attainment. Some efforts focus on a specific outcome, such as adoption of a policy, creation of a community resource, or completion of an infrastructure project. Even when the final goal is clear, it can still be useful to document intermediate progress, relationships, barriers, and lessons learned.
• Interactions. An evaluation may focus on how people communicate, collaborate, share decisions, respond to one another, or experience relationships within a program. For example, a family-support effort might examine patterns of communication and engagement rather than simply counting interactions.

These areas may relate either to program outcomes—what the program is intended to accomplish—or to process and implementation—how the work is planned and carried out.

Areas related specifically to program process and implementation may include:

• Planning. Who participated in planning? How were decisions made? Whose perspectives were included or missing? How well did the process reflect community priorities?
• Timeline. When did planning, implementation, evaluation, and follow-up begin? Were timelines realistic? If delays occurred, what contributed to them?
• Participation and engagement. How many people participated? How long did they remain involved? Were there patterns in who participated, who did not, or who stopped participating? What barriers or supports affected engagement?
• Methods. What strategies, activities, or methods were used? Were they implemented as intended, and what adaptations were made?
• Program implementation. What activities actually occurred? How often and for how long? Who participated? Where did activities take place? What resources were used? What changed during implementation, and why?

Whatever you choose to observe, define it carefully so that people collecting information understand what counts and what does not. Clear definitions, examples, and boundaries improve consistency and reduce the likelihood that different observers will interpret the same event in substantially different ways.

Returning to the health center example, the center might use activity logs or confidential self-reports to understand how often participants engaged in physical activity. Blood pressure could be measured using appropriate clinical procedures. Participants could report changes in well-being, energy, confidence, or barriers to activity through surveys or interviews. Follow-up interviews or activity logs could help determine whether participants continued physical activity after the program ended.

Decide how the observations will be conducted

Earlier, we introduced several methods of observation. Each has strengths, limitations, and ethical considerations.

• Direct observation. Direct observation may involve an evaluator, staff member, trained community observer, or other designated person observing defined activities or conditions. In program settings, participants should generally know that observation is occurring and understand its purpose. Observation of public settings may sometimes be conducted without identifying individual people, but privacy, ethics, local expectations, and applicable requirements should still guide the design.

Journals and activity logs can also support direct or participant observation. Observers may record what occurred, the context, relevant events, and their interpretations soon after an activity. When several people document the same period or activity, these records can provide multiple perspectives on how the program unfolded.

The format of journals or logs should fit the program and the people using them. Written journals may work well in some settings, while voice notes, structured forms, digital logs, or other accessible formats may work better in others.

• Participant observation. Participant observers take part in the setting or activity while documenting what they experience. They may be community members, program participants, staff, or evaluators who participate openly in activities. Their position can provide valuable insight into relationships, context, and experiences that may not be visible to an external observer.

For example, a small-grant program designed to support locally owned businesses might involve staff and community participants in workshops on budgeting, lending, business planning, and peer support. Staff and participants could reflect together on how the activities worked, which barriers emerged, and what participants found useful. In this setting, participant observation could contribute to learning while treating participants as partners in interpreting their own experiences rather than simply as subjects of observation.

• Self-reports. When relevant behaviors or experiences occur outside the program, participants may be the most appropriate people to describe them. Self-reports provide direct access to people's experiences, but responses can be influenced by memory, question wording, social expectations, concerns about privacy, or what participants believe the evaluator wants to hear. Clear questions, confidentiality protections, respectful data collection, and multiple sources of evidence can improve the usefulness of self-reported information.

Participants may benefit from clear guidance about what information is useful to record, especially when completing activity logs or journals. At the same time, evaluators should avoid coaching people toward preferred responses. Self-reporting is especially valuable because it allows participants to describe experiences, motivations, barriers, and changes that an outside observer may not be able to see.

Self-reports can be collected through individual and group interviews, focus groups, community meetings, surveys and questionnaires, journals, checklists, activity logs, or other ways for participants to share information.

• Reports from others. These may come from people who interact with participants or have relevant knowledge of the conditions being evaluated, such as educators, health professionals, service providers, family members, employers, or community partners. Consider whether the person has appropriate knowledge, whether participants have consented when necessary, and how the reporter's relationship, expectations, or biases may influence what they report.
• Electronic or technology-assisted observation. Technology can sometimes collect information that would be difficult to obtain otherwise. Examples include environmental sensors, activity monitors, clinical equipment, or traffic counters. Cameras, audio recording, location tracking, and other potentially intrusive technologies require especially careful consideration of informed consent, privacy, data minimization, data security, and who will have access to the information. Technology-generated data may appear objective, but equipment settings, algorithms, measurement error, and interpretation can still introduce limitations or bias.
• Tests or other assessment tools. Education, health, and human service organizations may use assessments to understand skills, knowledge, development, health status, or other outcomes. Whenever possible, use tools that are valid and appropriate for the population and purpose. Interpretation should also consider factors such as language, disability, culture, anxiety, fatigue, access to preparation, and testing conditions.
• Public and administrative records. Community-level indicators such as injury rates, health outcomes, school attendance, housing conditions, or transportation patterns may be available through government agencies, institutions, or other administrative systems. Consider the quality, timeliness, completeness, privacy protections, and limitations of these data before using them.

In the health center example, blood pressure could be measured at regular intervals using appropriate clinical equipment and procedures. Participants' physical activity could be documented through self-reported activity logs, interviews, or optional activity-tracking tools if participants choose to use them. Well-being could be assessed through participant reports using appropriate questions or validated measures.

Follow-up interviews, surveys, activity logs, or other appropriate methods could help determine whether participants continued regular physical activity after the program ended. Combining several types of information would provide a more complete picture than relying on any single measure.

Decide when you need to observe

Consider when information should first be collected and how often observations should occur. In many evaluations, collecting baseline information before or at the beginning of the program is important because it gives you a reference point for interpreting later changes.

Possibilities include:

• Before-and-after observation. Information is collected at the beginning and end of an evaluation period. This can show whether change occurred, but by itself may provide limited information about when change happened, why it happened, or which aspects of the program contributed to it.

For many evaluations, before-and-after measurement is most useful when combined with additional observations during implementation. Beginning with baseline information helps establish where participants, organizations, or community conditions started. Repeated measurement can then show whether change occurred gradually, rapidly, at particular stages, or only after the formal intervention ended.

Even when an effort has one highly visible goal—such as adoption of a policy or completion of a community project—evaluating only whether the final goal was reached leaves out valuable information. Documenting strategies, relationships, challenges, intermediate accomplishments, and adaptations can help explain what contributed to the result and what can be learned for future efforts.

• At regular intervals during the evaluation period. Observations might occur hourly, daily, weekly, monthly, or at another regular interval, depending on what is being measured. Regular schedules make it easier to compare information across time.
• At selected or randomly chosen intervals. Some observations may occur at varying times to capture a broader range of conditions or reduce the likelihood that the same circumstances are observed repeatedly. Random or strategically varied sampling can sometimes provide a more representative picture.
• At specific times or stages. You may want to examine conditions at particular stages of implementation or under different circumstances. For example, park use might be observed on weekdays and weekends, during different seasons, at different times of day, or during community events. A program evaluation might gather information during planning, outreach, implementation, evaluation, and follow-up.
• Ongoing observation. In some settings, staff, participants, sensors, logs, or other systems may provide ongoing information. Continuous collection should be used only when it is useful and proportionate to the evaluation need. Particularly when recording people or collecting digital data, consider whether continuous monitoring is necessary and how privacy will be protected.

At the health center, blood pressure might be measured at regular intervals during monthly meetings. Participants might document physical activity throughout the program using activity logs. Follow-up information could then be collected several months after the program to understand whether participants continued activities they found sustainable.

Define and describe the behaviors, products, conditions, and/or events that observers should be concerned with

Observers need clear definitions of what they are expected to document. The planning group—ideally including people who will conduct observations and people whose experiences are being represented—should establish shared definitions, examples, non-examples, and scoring or recording guidance for each element being observed. For example, an evaluation examining bullying or interpersonal aggression would need clear behavioral definitions rather than relying on assumptions about what those terms mean.

Design training for observers

Depending on their roles and previous experience, observers may need preparation in several areas:

• What should be recorded, and why. Observers should understand which contextual details matter, such as date, time, location, duration, environmental conditions, who was present, and other circumstances that might influence what occurred. Context can be as important as the event itself when interpreting observations.
• The definitions and descriptions of the behaviors, conditions, events, or situations being observed. Clear definitions improve consistency only if observers understand and practice using them.
• The effects of observation. People may behave differently when they know they are being observed. Observers should understand this possibility and use methods that reduce unnecessary influence while remaining transparent and respectful about the observation process.

Audio or video equipment can also affect behavior. Participants should understand what is being recorded, why it is being recorded, how the recordings will be used, who will have access, and how long they will be retained. Appropriate consent should be obtained before recording begins.

• Observer bias. Observers' relationships, expectations, experiences, identities, professional roles, and assumptions can influence what they notice and how they interpret it. This is especially important when staff members are evaluating programs in which they are invested. Training, reflection, structured observation tools, multiple observers, and opportunities to discuss differences in interpretation can help reduce the influence of bias. The goal is not to assume that observers can eliminate all perspective, but to recognize and manage how perspective can shape observation.
• Observer drift. Over time, observers may gradually interpret definitions differently from how they were originally established. Periodic review, refresher training, and comparison across observers can help maintain consistency.

Observational systems should include ways to identify and address observer effects, bias, inconsistent interpretation, or drift throughout the evaluation.

Devise checks for reliability and accuracy

Reliable information requires observers to apply definitions and measurement procedures consistently. Training is important, but it is also useful to periodically check whether different observers interpret the same behavior, condition, or event in similar ways. A participatory design process can strengthen this consistency because observers help define and understand what is being measured.

Ways to strengthen agreement and consistency include:

• Use a clearly defined standard. Observation criteria should describe the behaviors, conditions, or indicators being assessed and provide enough detail for different observers to apply them consistently. Checklists, coding guides, rating criteria, examples, and non-examples can support this process.

Research and evaluation teams often use standardized definitions and measurement procedures to improve consistency. Standards may include behavioral descriptions, clinical thresholds, environmental measurements, scoring rubrics, or other clearly defined indicators.

• Check inter-rater reliability. Inter-rater reliability refers to the degree to which different observers assess the same event or information consistently. Two or more observers may independently rate the same examples or observations and then compare results. The appropriate level of agreement depends on the type of measure and evaluation design; a single percentage threshold should not automatically be treated as adequate for every situation. If observers disagree, examine whether definitions, training, scoring procedures, or assumptions need clarification.
• Use periodic quality checks. A trained evaluator, supervisor, peer observer, or other designated person can periodically compare observations with those of regular data collectors. Repeated disagreement may signal that definitions, training, procedures, or measurement tools need to be reviewed.

Determine how to review and adjust your observational system for the next evaluation

Your observational system should itself be evaluated. Ask whether the system produced the information you needed, whether data collection was manageable, whether participants experienced the process as respectful, whether important perspectives or outcomes were missed, and whether any methods created unnecessary burden or privacy concerns. Use what you learn to improve the system for future evaluation cycles.

With thoughtful planning, appropriate training, clear ethical safeguards, and ongoing review, an observational system can provide useful information for understanding and improving your work.

In Summary

To evaluate a program or community effort effectively, you need a systematic way to collect useful information about what is being implemented, how people experience the work, what conditions influence it, and what outcomes occur. An observational system provides that structure. It may include direct or participant observation, self-reports, administrative records, assessments, technology-assisted measures, or other methods depending on the evaluation questions.

Observational systems should be feasible, participatory, ethically designed, and appropriate to the community and program context. Involving evaluators, data collectors, program staff, participants, and community members in design can improve the relevance of the information collected while helping identify concerns related to privacy, accessibility, interpretation, burden, and cultural context. Clear definitions, appropriate training, consistent procedures, and regular review can strengthen the reliability and usefulness of the resulting evaluation.

What do we mean by an observational system?

___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program

___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with

Why design an observational system?

___It can help you get reliable information

___It can help you find out exactly what you need to know, eliminating or reducing wasted effort

___It can ensure that observations are made

___It can make it easier to analyze your data

___It can help you avoid haphazard evaluation

___It will make it easier to justify your findings

___It can help you gain credibility with funders and policy makers

___It can let you pass on your practices with confidence

___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust

When should you design an observational system?

___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning

___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle

Who should design an observational system?

___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation

How do you design an observational system?

___Review your evaluation questions

___Decide what you need to observe to answer your questions:

• Participants’ behavior
• Someone else’s behavior
• Conditions
• Observations of results of behavior
• Participants’ knowledge or attitudes
• Someone else’s knowledge or attitudes
• Goal attainment
• Interactions
• Program process or implementation (e.g., number of participants)

___Decide how the observations will be conducted:

• Direct observation
• Participant observation
• Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
• Second-hand reports, including interviews, journals, surveys, etc.
• Electronic or mechanical observation
• Tests or other similar observation tools
• Public records and the like for community-level indicators

___Decide when you need to observe:

• Pre- and post- observation
• At regular intervals during the evaluation period
• At irregular intervals during the evaluation period
• At specific times during the evaluation period
• Continuously

___Define and describe the behaviors, conditions, and/or events that observers should be concerned with

___Train observers in:

• What it’s important to record, and why
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
• Effects of observation
• Observer bias
• Observer drift

___Devise checks for reliability and accuracy:

• Use an external standard
• Check for inter-rater reliability
• Use random third-party checks

___Adjust the system for the next evaluation

When you hear the word “experiment,” you may picture people in laboratory coats working with specialized equipment. In a broader sense, however, an experiment simply involves trying something and systematically examining what happens. It can be as simple as testing a different ingredient in a recipe or as complex as developing and evaluating a comprehensive effort to improve child health outcomes across a city or state.

Researchers in public health and the social sciences conduct studies to understand how programs, policies, environments, and other conditions influence behavior and outcomes. Community programs and interventions can also be viewed as opportunities for systematic learning. They often begin with an assumption—or theory of change—about how particular activities are expected to contribute to desired outcomes. Those assumptions should be examined rather than taken for granted. Evaluation helps determine whether change occurred, how the program may have contributed to it, for whom it worked, under what conditions, and what might be improved.

In this section, we will examine several ways to structure an evaluation so that you can understand whether and how your program is contributing to change. These structures are known as experimental, quasi-experimental, or evaluation designs.

What do we mean by a design for the evaluation?

Every evaluation is a form of systematic inquiry. You may want to understand how effective your program or effort is overall, which components are working well and which should be adjusted, whether outcomes differ among groups, or how particular conditions influence results. To answer those questions meaningfully, the evaluation needs a structure that connects the questions you ask with the information you collect and the conclusions you draw. That structure is the evaluation design.

The design depends on the kinds of questions your evaluation is meant to answer.

Common evaluation questions include:

• Does a particular program, intervention, policy, or change in access and opportunity contribute to changes in behavior, physical or social conditions, health or development outcomes, or other indicators of success?
• Which components or elements of the program appear to contribute most strongly to the outcomes?
• What intended or unintended effects occurred, and how did they influence participants, communities, or outcomes?
• What happens when you introduce or adapt a new method, activity, or policy?
• Will an approach that worked in another setting also work in your community, with your participants, and under your local conditions?

To answer questions like these, you need a design that helps distinguish between changes associated with your program and changes that may have resulted from other factors. The stronger and more appropriate the design, the more confidence you can have in the conclusions you draw. Later in this section, we will discuss several ways to structure an evaluation.

Why should you choose a design for your evaluation?

Evaluation may seem straightforward: if conditions improve, the program appears to be working; if they do not, something should change. In practice, evaluation is more complicated. How will you define and measure progress? If outcomes do not improve, which parts of the program should change? If improvement occurs, how confident are you that the program contributed to it rather than another factor? And even when a program is working well, which components are essential and which could be strengthened, adapted, or removed? A thoughtful evaluation design helps answer questions like these.

Reasons to design your evaluation carefully include:

• To strengthen the reliability and usefulness of your evaluation. A well-designed evaluation can provide information you can use with greater confidence. It can help you understand whether change occurred and how strongly the evidence supports the conclusion that the program contributed to that change.
• To identify strengths and areas for improvement. A good design can help show which program components appear effective, where barriers exist, and where adjustments may increase effectiveness, accessibility, or equity.
• To strengthen the credibility of your findings. Clearly designed and transparently conducted evaluations make it easier for participants, community partners, funders, policymakers, and others to understand and assess the evidence.
• To identify factors outside the program that may influence outcomes. Participants' prior experiences, community events, policy changes, economic conditions, changes over time, and many other factors can affect outcomes. A strong design can help identify these influences and clarify how they should be considered when interpreting results.
• To identify unintended consequences. Programs may produce outcomes that were not anticipated, including benefits, harms, inequities, or changes in areas that were not originally measured. A thoughtful design makes it more likely that these effects will be recognized and addressed.
• To provide a clear plan for the evaluation. An appropriate design helps determine what information should be collected, when it should be collected, from whom, and how it should be analyzed. Planning this structure in advance can improve both efficiency and the quality of the resulting information.

When should you choose a design for your evaluation?

Once you have identified your evaluation questions and gathered information about the issue, community context, and possible approaches, the next step is to choose an evaluation design. Ideally, evaluation planning begins alongside program planning. Evaluation should be an integral part of your program rather than something added only after implementation has begun.

In practice, community organizations, public agencies, and other programs may begin formal evaluation months or even years after implementation because of limited staff time, funding, evaluation capacity, or other constraints.

Even when evaluation begins later, the same basic process applies: clarify your questions, gather relevant information, choose a feasible design, and follow the evaluation process systematically. If your program operates in cycles, it may be useful to begin evaluation at the start of a new cycle or at the beginning of participants' involvement so that baseline information can be collected.

Programs with rolling enrollment or continuous services may require a different approach. You might evaluate outcomes among newly enrolled participants, follow different groups over time, or begin examining the program as it currently operates. The design should reflect the structure of the program and the questions you are trying to answer.

Who should be involved in choosing a design?

The Community Tool Box generally recommends a participatory process that includes people affected by the program as well as staff, community partners, evaluators, researchers, and other stakeholders. Evaluation design benefits from this same participation, while also requiring appropriate methodological expertise.

Research and evaluation design can involve complex questions about measurement, bias, comparison groups, statistical analysis, ethics, and interpretation. For this reason, someone with relevant evaluation or research experience should usually help coordinate the technical aspects of the design. That person or team should work collaboratively with participants and community partners rather than making design decisions in isolation. Researchers can explain the strengths, limitations, resource requirements, and ethical considerations of different designs, while community partners can help determine which approaches are appropriate, feasible, and meaningful in the local context.

How do you choose a design for your evaluation?

How do you decide which research design will best serve the purposes of your evaluation?

Consider four areas:

• The nature of the evaluation or research questions you are trying to answer
• Potential challenges to drawing valid conclusions and ways to reduce those challenges
• The research designs that may be appropriate and what each one requires
• Whether a particular design fits your program structure, participant preferences and consent, ethical requirements, timeline, and available resources

We will begin by examining the concerns a research design should address, then consider several common designs and their strengths and limitations, and finally discuss how to select a design that is both feasible and capable of providing useful information.

Note: In this part of the section, we discuss evaluation as a type of research. For that reason, we sometimes use the term “research” even when the same principle could be described as “evaluation.”

Concerns research designs should address

One of the most important considerations in designing research or evaluation is whether the design will provide information that supports the conclusions you want to draw. A weak design may show that something changed without helping you understand why it changed. For evaluation, that can make it difficult to know what should be continued, adapted, or improved.

Consider a simple example. Suppose you measured children's height at age six, encouraged them to eat carrots regularly for three years, and measured them again at age nine. Most would probably be taller. Without an appropriate comparison or other evidence, however, you could not reasonably conclude that carrots caused the increase in height. Children ordinarily grow as they age, and the design would not distinguish that natural development from any effect of the intervention.

Two broad types of validity are especially important in research design: internal validity, which concerns whether the intervention actually contributed to the observed change, and external validity, which concerns how well findings may apply to other people, places, or circumstances.

Threats to internal validity

Threats to internal validity are alternative explanations for the changes observed during an evaluation. Other events, participant characteristics, measurement changes, or natural processes may occur at the same time as the program. If the evaluation cannot distinguish among these influences, it becomes harder to determine how much of the observed change can reasonably be attributed to the program.

Common threats to internal validity include:

• History. Events that occur before or during the evaluation may influence outcomes. These might include participants' prior experiences as well as external events such as policy changes, economic shifts, community conflict, natural disasters, public health emergencies, or other major developments.
• Maturation. People and communities change naturally over time. Children develop as they grow, people gain experience, physical conditions may change with age, and participants may become tired or more comfortable with activities as time passes. These changes can occur independently of the program.
• The effects of measurement or observation. Being surveyed, tested, recorded, observed, or repeatedly asked about a behavior can itself influence how people think or act.
• Changes in measurement. Equipment, assessment tools, data systems, or observers may change over time. Different instruments or observers may also produce different results. Consistent procedures and quality checks can reduce these problems.
• Regression toward the mean. Extremely high or low measurements often move closer to the average when measured again, even without an intervention. This can be particularly important when programs intentionally enroll people because they have unusually high or low values on a particular measure.
• Participant selection. People who participate in a program may differ in important ways from those who do not. Selection procedures may also unintentionally favor people who already have greater access to resources, fewer barriers, or characteristics associated with stronger outcomes. For example, an employment program rewarded only for job placements might be encouraged to enroll people who are already closest to employment while excluding people who face greater barriers. Self-selection can also matter because people who voluntarily enroll may already be particularly motivated to make a change.

Selection becomes especially important when two or more groups are compared. If the groups differ substantially before the intervention begins, differences observed later may reflect those initial differences rather than the program itself.

• Loss of data or participants. Missing information and participant attrition can affect results, particularly when people who leave a study or program differ from those who remain. When comparing groups, different rates or patterns of attrition can make the comparison less reliable.
• The nature and timing of change. Change is rarely perfectly steady. Outcomes may improve, decline, fluctuate, or take time to emerge. Measurements taken over too short a period or at poorly chosen times may not capture the actual pattern of change.
• Combinations of multiple factors. Several threats to validity may occur at the same time. For example, an educational program tested only among students who already have extensive experience learning multiple languages would involve both selection effects and differences in prior experience.

Threats to external validity

External validity concerns how well findings may apply beyond the people, setting, and circumstances included in the evaluation. An intervention that works well in one setting may not produce the same results elsewhere if the populations, resources, institutions, cultures, implementation conditions, or other contextual factors differ.

Threats to external validity, sometimes discussed in terms of generalizability or transferability, often arise from interactions between the intervention and the particular conditions under which it was implemented.

Examples include:

• Interaction between data collection and the intervention. Baseline testing, surveys, or observation may influence how participants respond to the program. If another setting does not use the same data-collection procedures, outcomes may differ.
• Interaction between participant selection and the intervention. A program may work particularly well for people who voluntarily enroll, are highly motivated, or meet specific eligibility criteria, but those findings may not apply equally to people with different experiences or circumstances.

For example, people who participate in a parenting program because of a court or child-welfare requirement may experience the program differently from people who enroll voluntarily. Differences in motivation, stress, trust, available supports, and other circumstances may influence outcomes and should be considered when comparing results.

• The effects of the research arrangements. People may respond differently because they know they are being observed or because of their relationship with particular researchers, facilitators, or staff members.

The broader lesson is that the act of observation can sometimes influence the behavior being observed. Evaluators should consider whether data-collection procedures themselves may be contributing to the outcomes they see.

• Exposure to multiple programs, services, or interventions. Participants may be involved in several programs or experience multiple changes at the same time. These combined influences can make it difficult to determine the effect of one intervention in isolation.

For example, an adult might participate in a high school completion program while also receiving substance use recovery services. A person managing diabetes might receive medication while participating in nutrition, physical activity, or peer-support programs. These overlapping services may reinforce one another, interfere with one another, or make it difficult to identify the contribution of any single component.

Common research designs

Research design is a large field, and this section provides only an introduction. Evaluation designs differ in several important ways, including how many times outcomes are measured, when measurements occur, whether one or multiple groups are included, and how participants or groups are selected.

Program evaluations often seek to answer three basic questions:

• Did meaningful change occur during the evaluation period?
• To what extent did the program, intervention, or effort contribute to that change?
• What program components, contextual conditions, or outside influences helped produce or prevent the change?

Outcomes may be influenced by the program, by outside factors, or by both. Participants' experiences, natural development, policy changes, economic conditions, organizational changes, community events, and other influences may all affect results. Understanding those influences can help you improve the program and interpret outcomes more accurately.

Measuring conditions only before and after a program can show whether something changed, but it may not explain why. For that reason, evaluation designs should consider potential threats to validity as well as practical and ethical constraints.

Research designs often differ according to the number and timing of measurements and whether one or multiple groups are included. We will begin with single-group designs and then consider designs involving multiple groups.

Before continuing, it is useful to define several basic research terms.

Researchers commonly refer to measurements or observations collected before an intervention begins as baseline measures or baseline observations. Baseline information establishes a starting point against which later observations can be compared.

Other useful research terms include:

• Independent variables are the program, intervention, policy, method, or condition whose effects you want to examine. An evaluation may include one independent variable or several.
• Dependent variables are the outcomes or conditions expected to change in response to the independent variable. These might include behaviors, health outcomes, community conditions, knowledge, attitudes, or other indicators.
• Measures are the tools or procedures used to assess those outcomes. They may include community assessments, administrative data, observations, surveys, interviews, validated scales, clinical measurements, or counts of particular events or conditions.
• Observations may involve formal measurement or systematic documentation of what occurs in a particular setting. In research, the term is sometimes used broadly to describe many forms of measurement and data collection.

Pre- and post- single-group design

One of the simplest designs is a pre- and post-measurement design. You measure an outcome before the intervention, implement the program, and measure the same outcome again afterward. Examples might include unemployment rates, environmental pollution levels, health indicators, or participant knowledge.

This design can show whether change occurred between the beginning and end of the evaluation period, but by itself it usually cannot establish why the change occurred. Other events, natural trends, changes in measurement, or participant characteristics may explain some or all of the difference. Even so, collecting information before and after an intervention generally provides more useful information than having no baseline at all.

Consider blood pressure in a cardiovascular health program. A lower measurement at the end of the program than at the beginning does not show what happened between those two points. Blood pressure may have fluctuated throughout the evaluation period, measurement conditions may have changed, or other health or life changes may have influenced the result. Repeated measurement can provide a clearer picture.

Interrupted time series design with a single group

An interrupted time series design uses repeated measurements before and after the introduction of an intervention. Establishing a pattern before implementation provides a baseline trend, while repeated measurements afterward help show whether that pattern changed when the intervention began.

This approach can provide substantially stronger evidence than a simple pre- and post-design because it shows how outcomes change over time. If the outcome begins changing before the intervention starts, for example, that suggests that another factor may be contributing to the trend.

Some time-series designs also examine the effects of introducing different program components at different times. When appropriate and ethically feasible, this can help evaluators understand which components appear associated with particular changes.

Many variations of interrupted time series designs are possible, including changes in measurement frequency, staggered implementation, or evaluating combinations of program components.

When using a time-series design, it is important to examine outcomes closely enough to identify meaningful patterns. An overall average can sometimes hide changes occurring in particular periods, locations, or groups. Disaggregating information when appropriate can reveal effects that would otherwise be missed.

Interrupted time series design with multiple groups

A multiple-group time series design applies similar repeated measurements across two or more groups, sites, or communities. The intervention may be introduced at different times across groups, creating multiple baselines that can strengthen confidence that observed changes are associated with the intervention rather than a single external event.

This type of design may allow you to:

• Examine the same method or program across two or more groups or sites
• Examine whether an approach produces similar or different results across populations or settings
• Vary the timing or intensity of an intervention when doing so is appropriate
• Compare different interventions or implementation approaches
• Examine whether the order or sequence of program components influences outcomes

Many other variations are possible, depending on the program and evaluation questions.

Control or comparison group design

Another common approach is to compare outcomes between a group receiving the intervention and another group that does not receive the same intervention during the same period, receives usual services, or receives a different approach. A well-designed comparison can help distinguish program effects from changes that would have occurred anyway.

A control group is generally part of a design intended to isolate the effect of the intervention by holding other influences as similar as possible. A comparison group may be selected because it resembles the intervention group on relevant characteristics, even when participants are not randomly assigned.

Randomized controlled designs are among the strongest methods for estimating causal effects when random assignment is feasible, ethical, and appropriate. In these designs, eligible participants or groups are assigned by chance to different study conditions. Randomization helps reduce systematic differences between groups before the intervention begins, although it does not eliminate every possible limitation or guarantee that findings will generalize to other settings.

For example, eligible participants might be randomly assigned to receive either a new program or an existing standard service. If implementation and measurement are otherwise comparable, differences in outcomes between groups can provide strong evidence about the effect of the different approaches.

Community organizations may not always be able—or ethically justified—to withhold a potentially beneficial program solely for evaluation purposes. Alternatives may include wait-list designs, phased implementation, usual-services comparisons, matched comparison groups, or other quasi-experimental approaches. The evaluation should not create avoidable harm or unfairly restrict access to needed services simply to strengthen the research design.

When a program has limited capacity and already uses a wait list, phased or randomized enrollment may sometimes create a comparison while preserving eventual access to services. Another option is to provide participants waiting for the full program with existing resources, alternative services, or another appropriate form of support rather than leaving them without assistance solely for research purposes.

When no appropriate control or wait-list group exists, organizations may use a comparison group from another program, community, or setting. Because those groups may differ in important ways, evaluators should examine baseline characteristics carefully and be cautious about attributing differences in outcomes solely to the intervention. In some situations, a strong interrupted time series design may provide more credible information than a weak comparison between substantially different groups.

Groups may appear similar while differing in ways that influence outcomes. For example, participants who voluntarily enter a substance use treatment program may differ from people who are required to participate through a court or institutional process. Motivation, trust, available supports, prior treatment experiences, and other circumstances may influence results independently of the treatment model.

Average results can also hide important differences within a single group. An intervention that appears only modestly effective overall may work particularly well—or poorly—for participants with particular circumstances, experiences, or barriers. Examining variation in outcomes can provide important information for improving equity and effectiveness.

Choosing a design

This section provides only an introduction to evaluation design. Research methods include many additional experimental, quasi-experimental, observational, qualitative, mixed-methods, implementation, and participatory approaches. Statistical methods can also help address some design limitations, although analysis cannot fully compensate for poor-quality or inappropriate data.

When possible, seek technical assistance from people with evaluation expertise. Universities, research centers, public agencies, community evaluation organizations, funders, and consultants may be able to provide methodological support. Whatever design you use, community members and program participants should remain involved in identifying meaningful questions, outcomes, ethical concerns, and interpretations.

Your goal is to obtain the most credible and useful information possible given your evaluation questions, program structure, ethical responsibilities, participant preferences and consent, timeline, and resources. A community evaluation does not need to resemble a laboratory experiment to be valuable. It does need to be designed carefully enough that you understand what the evidence can and cannot tell you.

Consider your evaluation questions

What do you need to know? If your question is simply whether a particular condition changed, a pre- and post-design may sometimes be sufficient. If you want to know whether the program contributed to that change, you will need a design that provides stronger evidence about alternative explanations.

Comparison groups can be useful, but they are not the only way to strengthen an evaluation. Repeated measurements, multiple forms of evidence, qualitative information, implementation data, and careful documentation of external events can provide a much fuller picture. For many community organizations, an interrupted time series design may offer a practical balance between rigor and feasibility.

If your program already includes multiple groups, sites, or classes working toward similar goals, you may be able to compare outcomes across them or stagger implementation in ways that provide additional information. Any such approach should be developed with attention to fairness, participant expectations, and whether differences in implementation are appropriate.

Sometimes the central question is not simply whether a program works, but whether it works better than another available approach. Employment training, parenting support, chronic disease management, conflict resolution, and many other programs may raise this question. Depending on the setting, comparisons may involve another program, usual services, published research, administrative information, or community-level indicators.

Some community efforts have outcomes that can be measured relatively directly. An initiative focused on reducing water pollution, for example, may use repeated water-quality measurements as a central outcome. Even in these situations, however, you may also want to understand which activities contributed to improvement, whether benefits were distributed equitably, and whether changes were sustained over time.

Consider the nature of your program

What does the program look like, and what is it intended to accomplish? Does it work with participants individually or in groups? Does it operate in fixed cycles, such as classes or workshops, or can people enter at different times? How much depends on staff implementation, participant activity, partnerships, organizational practices, or the surrounding environment? How important are relationships, physical setting, accessibility, language, organizational culture, and other contextual factors?

• If participants take part in distinct groups or sites, a multiple-group design may sometimes be practical. If participation is primarily individual, a single-group longitudinal or time-series design may be more appropriate.
• If your program is time-limited, choose a design that fits the program schedule and allows enough time to capture meaningful outcomes. If multiple program cycles occur, lessons from earlier cycles may inform adaptations to later ones, but those changes should be documented clearly.
• If your program has no clear beginning or end, you may need to follow participants from their individual starting points or organize analysis around different levels of exposure, participation, or baseline conditions.

Community programs operate in changing environments. People entering a program at different times may encounter different staff, participants, community events, resources, policies, or organizational conditions. These differences can influence experiences and outcomes even when the formal program model remains the same. Evaluators should document important contextual changes and consider them when interpreting results.

• If organizational culture, relationships, or program context are central to the intervention, include measures that capture those factors rather than treating them as background conditions. Comparisons across different organizational settings may also be useful when they are appropriate and sufficiently similar.

Many evaluation designs can be adapted to different situations. The important goal is to select an approach that fits both the evaluation questions and the realities of the program.

Consider what participants and staff will consent to

Evaluation methods should respect privacy, dignity, autonomy, safety, and informed participation. Extensive observation, recording, repeated testing, or collection of sensitive information can create burdens or risks for participants and staff. A technically strong design is not appropriate if it depends on methods that are unnecessarily intrusive or unacceptable to the people involved.

Privacy and safety concerns may be especially important when participation in a program involves sensitive health information, immigration status, substance use, interpersonal violence, legal issues, or other information that could create harm if disclosed. Participants should not be pressured to provide information that is not necessary for the evaluation.

Explain what information will be collected, why it is needed, how it will be used, who will have access to it, what protections will be in place, and what choices participants have. Include participants and staff in designing data-collection procedures whenever possible. Their concerns may reveal risks or burdens that evaluators would otherwise overlook. :chatgpt-content-reference{index="1"}

Consider your time constraints

Choose a design that can provide credible information within the time available. The goal is not necessarily to create the most complex design possible, but to select an approach that is appropriate to the questions and transparent about its limitations.

Common time constraints include:

• Program structure. Evaluation may need to align with a regular program cycle, school year, funding period, or implementation schedule.
• Funding. Grants and contracts may establish deadlines for reporting or evaluation. Pilot programs may also have limited time to demonstrate early outcomes, even when longer-term effects will take more time to appear.
• Participants' schedules and responsibilities. Work, school, caregiving, transportation, seasonal employment, cultural or religious commitments, and other responsibilities may affect when people can participate in program or evaluation activities.
• The urgency of the issue. When a program addresses serious risks to health or safety, evaluation timelines should support timely decisions rather than delaying necessary action solely for methodological reasons.
• The availability of evaluation support. Evaluators, analysts, community partners, and other people involved in the evaluation may only be available during particular periods.

Consider your resources

Resources include funding, staff time, community knowledge, partnerships, technology, space, equipment, data systems, and evaluation expertise. A lack of one resource can sometimes be addressed through another—for example, by partnering with a university, public agency, community organization, or evaluator who can provide technical assistance.

Begin by identifying the design that best fits your questions and ethical responsibilities, then determine how to make that design feasible. You may need to simplify or adapt it, but understanding the strongest realistic option can help you make intentional compromises rather than automatically choosing the least resource-intensive design.

In Summary

Evaluation design strongly influences how confidently you can interpret your findings and how useful those findings will be for improving a program or intervention. A strong design considers threats to internal validity—whether the intervention contributed to the observed change—as well as external validity or transferability—whether findings may apply in other settings or populations.

Designs such as pre- and post-measurement, interrupted time series, multiple-group designs, comparison groups, and randomized controlled designs each have different strengths, limitations, ethical considerations, and resource requirements. They can often be adapted or combined to fit the needs of community programs.

A useful evaluation design should address your questions while also reflecting the nature of the program, community context, participant and staff consent, ethical responsibilities, available time, and resources. The goal is not simply to create the most rigorous design possible, but to generate credible, relevant, and actionable information in a way that respects the people and communities involved.

What do we mean by a design for the evaluation?

___The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know

___An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors

Why should you choose a design for your evaluation?

___So your evaluation will be reliable

___So you can pinpoint areas you need to work on, as well as those that are successful

___So your results are credible

___So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants

___So you can identify unintended consequences (both positive and negative) and correct for them

___So you’ll have a coherent plan and organizing structure for your evaluation

When should you choose a design for your evaluation?

___In the ideal, when you’re planning the program and evaluation, before you start any implementation

___In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters

Who should be involved in choosing a design for your evaluation?

___It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails

How do you select an appropriate design for your evaluation?

___Take into consideration the necessary research considerations

• Understand threats to internal validity ( whether the intervention produced the change)
  • History
  • Maturation
  • The effects of testing or observation on participants
  • Changes in measurement
  • Regression toward the mean
  • The selection of participants
  • The loss of data or participants
  • The nature of change
  • A combination of the effects of two or more of these
• Understand threats to external validity (generalizability of your findings)
  • Interaction of testing or data collection and the program or intervention
  • Interaction of selection procedures and the program or intervention
  • The effects of the research arrangements
  • The interference of multiple programs or interventions

___Understand common research designs

• Pre- and post- single group design
• Interrupted time series design with a single group (simple time series)
• Interrupted time series with multiple groups (multiple time series)
• Control or comparison group

___Choose a design

___Consider your evaluation questions

___Consider the nature of your program

___Consider what your participants and staff will consent to

___Consider your time constraints

___Consider your resources

In previous sections of this chapter, we discussed studying the issue, choosing a research design, and creating an observational system for gathering information for your evaluation. The next step is to collect and analyze your data—examining what the information shows and what it may mean for your work. This section focuses on how to carry out that process.

What do we mean by collecting data?

Collecting data means putting your information-gathering plan into practice. You have already decided how you will obtain information—through direct observation, interviews, surveys, assessments, administrative records, or other methods—and now you and others involved in the evaluation need to carry out that plan. Effective data collection also requires clear definitions, appropriate timing, consistent procedures, and careful documentation so that the information you collect is useful for answering your evaluation questions.

Recording and organizing information may take different forms depending on the type of data you are collecting. Your data-collection procedures should also reflect how you plan to analyze and use the information. Whenever possible, information should be recorded during data collection or soon afterward so that important details are not lost. Data should also be stored securely and in ways that protect participants' privacy and confidentiality.

Some of the steps involved in preparing collected information may include:

• Bringing together information from all relevant sources and observations
• Creating secure backups of forms, records, recordings, and other materials to reduce the risk of loss, accidental deletion, or damage
• Entering narratives, numbers, and other information into appropriate software, databases, spreadsheets, or data-management systems
• Preparing quantitative information for analysis. This may include entering observations into tables or spreadsheets and calculating descriptive statistics such as the mean, median, mode, percentages, rates, or other relevant measures
• Transcribing relevant portions of audio or video recordings when text versions are needed for analysis
• Coding qualitative information by identifying categories, concepts, themes, or other features that make the information easier to organize and analyze
• Organizing data in ways that support your evaluation questions. You might group information by outcome, participant or group, time period, activity, location, implementation stage, or another relevant category. Looking at data in more than one way can also help identify relationships and differences that might otherwise be missed.

Research often distinguishes between independent variables and dependent variables. An independent variable is the program, intervention, policy, method, or condition whose effects you want to examine. A dependent variable is an outcome or condition that may change in response to that intervention. For example, a smoking-cessation program may be treated as the independent variable, while changes in participants' smoking behavior may be one of the dependent variables.

What do we mean by analyzing data?

Analyzing information means examining it systematically to identify patterns, relationships, trends, differences, and other findings that can help answer your evaluation questions. Analysis may involve statistical procedures, comparisons across groups or time periods, visual inspection of trends, or careful interpretation of interviews, observations, documents, and other qualitative information. The purpose is to develop the most accurate and useful understanding possible of your program, its implementation, its outcomes, and the conditions that may have influenced those outcomes.

Evaluations commonly use two broad types of data, although not every evaluation will include both. Quantitative data are expressed as numbers and can be summarized or analyzed mathematically. Qualitative data include descriptions, experiences, observations, quotations, interpretations, and other information that helps explain meaning and context. Each provides different kinds of insight, and they often work especially well when used together.

Quantitative data

Quantitative data are often collected directly as numbers. Examples include:

• The frequency, rate, duration, or intensity of particular behaviors, events, or conditions
• Assessment or test scores related to knowledge, skills, health, or other outcomes
• Survey responses, such as ratings of satisfaction, stress, confidence, access, well-being, or reported behavior
• Counts, rates, or percentages describing a population—for example, the percentage of people diagnosed with a particular health condition, the unemployment rate, languages used in a community, age distribution, or levels of educational attainment

Information collected in other forms can sometimes be converted into quantitative data. Researchers might count how often particular themes or events appear in interviews or records, for example, or use a rating scale to describe the intensity or frequency of an observed condition. Community initiatives may also document the number and type of environmental, organizational, programmatic, or policy changes connected to their work. Whether converting information into numbers is useful depends on what you are trying to understand and whether doing so preserves the meaning of the original information.

Quantitative information may be summarized with descriptive statistics such as frequencies, percentages, rates, averages, medians, and ranges. More advanced statistical procedures can examine differences between groups, changes over time, associations among variables, and the strength of evidence for particular conclusions. Numerical analysis can be powerful, but numbers do not interpret themselves. The quality of the findings still depends on what was measured, how measurements were collected, whether the sample represents the people or conditions of interest, and whether the analysis is appropriate for the evaluation design.

Qualitative data

Qualitative information provides detail about people's experiences, meanings, perspectives, interactions, and the context in which events occur. It can help explain findings that numerical measures alone may not capture. For example, a test score may indicate how a student performed, while an interview or observation may help explain the student's experience, confidence, barriers, learning environment, or response to the result.

Qualitative information can sometimes be summarized numerically—for example, by counting how often themes appear or asking participants to rate dimensions such as satisfaction, importance, accessibility, or usefulness. However, converting qualitative information into numbers can reduce some of its context and meaning.

Rating scales illustrate this limitation. Two people may both select the same numerical rating while having very different reasons for doing so. A participant who rates a program a “3” might appreciate its content but find the schedule difficult, while another might like the schedule but feel the content does not meet their needs. The number alone cannot explain those differences.

Qualitative information can help explain why particular methods work well or create barriers, how participants experience a program, whether an approach fits the local context, and what participants consider most important. It can also reveal patterns or factors that were not anticipated when the evaluation was designed.

Whenever appropriate, collecting both quantitative and qualitative information can provide a more complete understanding of a program and its outcomes.

Neither quantitative nor qualitative analysis is completely free from human judgment. Quantitative analysis uses standardized mathematical procedures, but decisions about what to measure, which people or data to include, which statistical procedures to use, and how findings are interpreted still affect the results. Qualitative analysis involves interpretation more directly, making it especially important to use clear procedures, consider multiple perspectives, document how conclusions were reached, and reflect on how researchers' assumptions and experiences may influence interpretation.

Why should you collect and analyze data for your evaluation?

Not every organization has the resources to conduct a large or highly technical evaluation. Smaller community-based organizations may use simpler evaluation approaches that still provide valuable information for understanding and improving their work. Regardless of the level of complexity, systematic data collection and interpretation can strengthen decision-making and future planning.

• Data can show whether meaningful changes occurred in the outcomes you hoped to influence. Collecting information before, during, and after an intervention can help you determine whether conditions changed and whether the pattern of change is consistent with the goals of the program.

In statistical analysis, the term statistical significance has a specific meaning. Researchers often use a threshold such as p < .05 to indicate that the observed result would be relatively unlikely under a particular statistical assumption if there were actually no effect or difference. Statistical significance should not be confused with practical importance, the size of an effect, or the percentage of participants who benefited.

Statistical results should therefore be interpreted together with the evaluation design, effect size, confidence intervals when available, sample size, data quality, practical significance, and other evidence. A statistically significant result does not by itself prove that a program caused a particular outcome.

• Analysis can identify factors associated with differences in outcomes. For example, participants who have access to a peer-support group may show different outcomes from participants who do not. Findings like these can help identify implementation features that deserve closer examination.
• Analysis can identify relationships among different factors. Statistical procedures may reveal correlations or associations among variables. Two outcomes might increase together, decrease together, or move in opposite directions. These relationships can generate useful questions, but correlation alone does not establish that one variable causes another.
• Analysis can help explain why your work was more or less effective than expected. Combining quantitative and qualitative information can help identify implementation strengths, accessibility barriers, contextual conditions, participant experiences, or other factors that influence outcomes.
• Analysis can provide credible evidence for stakeholders. Participants, community members, governing boards, funders, and other partners may want to understand what was accomplished, what challenges remain, and what the organization has learned. Evidence about implementation, intermediate outcomes, and longer-term outcomes can support accountability and future planning.
• Regular use of data demonstrates a commitment to learning and improvement. Reviewing information about progress, barriers, and outcomes can help organizations act as responsible stewards of community resources.
• Sharing what you learn can strengthen work beyond your own program. Findings about effective strategies, implementation challenges, adaptations, and community context may help other organizations develop or improve related efforts.

When and by whom should data be collected and analyzed?

Whenever possible, data collection should begin before or when the program starts so that you can establish baseline information. Collecting information before implementation can also reveal trends that were already occurring. Data collection may continue throughout the program and, when longer-term outcomes matter, during an appropriate follow-up period after the formal intervention ends.

Data can be analyzed at different points depending on the purpose of the evaluation. A primarily summative evaluation may emphasize analysis after the main data-collection period so that findings can be considered together. A formative evaluation may examine information throughout implementation so that emerging findings can inform adjustments. Many evaluations combine these approaches by reviewing data periodically while also conducting a more comprehensive analysis at the end of an evaluation cycle.

Who collects and analyzes information depends on the methods being used, available expertise, resources, ethical requirements, and the degree of participation built into the evaluation.

Possible approaches include:

• Partnering with or hiring an external evaluator, researcher, university program, or other person with relevant expertise
• Conducting a less technical but systematic evaluation using descriptive statistics, participation records, interviews, feedback, and other information that can still provide useful insight into the program
• Building staff or community evaluation capacity through training, courses, technical assistance, or mentoring
• Collecting information internally and partnering with someone who has appropriate expertise to conduct or review more specialized analysis
• Using qualitative methods when they are appropriate to the evaluation questions. Qualitative information may be especially valuable for understanding participant experiences, implementation, accessibility, barriers, and context, even when it cannot by itself answer every question about program effectiveness
• Using an appropriate comparison strategy when your design and ethical circumstances support one. Comparisons can strengthen an evaluation, but conclusions should still take into account baseline differences, sample size, uncertainty, and other factors rather than relying only on raw percentages.

In a participatory evaluation, community members, program participants, staff, and other stakeholders can contribute to both data collection and interpretation. Some forms of measurement may require specialized training, while other information may be best collected or interpreted by people with lived experience or deep knowledge of the community. Even when statistical analysis is conducted by a specialist, community participants can play an important role in interpreting what the findings mean, identifying missing context, and deciding how the results should inform future action.

How do you collect and analyze data?

Whether your evaluation uses highly formal research methods or a simpler community-based approach, several basic steps can help you collect and analyze information systematically.

Implement your measurement system

We've previously discussed designing an observational system to gather information. The next step is to put that system into practice.

• Clearly define and describe the measurements or observations you need. Definitions should be specific enough that different data collectors understand what is being measured and record information consistently.
• Select and prepare data collectors. Depending on the evaluation, they may need training in observation, interviewing, confidentiality, data security, accessibility, informed consent, or inter-rater reliability.
• Collect information at appropriate times and for an appropriate period. This may include reviewing existing records, conducting interviews, administering surveys, facilitating focus groups, observing program activities, or gathering other forms of data.
• Record information using agreed-upon procedures. Depending on the setting, this may involve secure digital forms, databases, spreadsheets, written records, audio or video recordings, journals, activity logs, or other appropriate formats.

Organize the data you've collected

How you organize information depends on your evaluation questions and how you plan to analyze it.

• Enter and store information in an appropriate system. This might include a database, spreadsheet, qualitative analysis program, secure document system, or GIS (Geographic Information Systems) platform. Use appropriate privacy and security protections, particularly when data include personally identifiable or sensitive information.
• Transcribe audio or video recordings when needed. Transcription can make recorded information easier to search, code, compare, and analyze. Review transcripts for accuracy and protect identifying information appropriately.
• Score assessments or tests according to the procedures appropriate for the instrument and record the results accurately.
• Sort information in ways that support your questions. You might organize data by participant group, outcome, activity, location, time period, implementation stage, demographic characteristic, or another relevant category.
• When useful and appropriate, summarize qualitative information quantitatively. This might include counting how often particular themes, experiences, or barriers are mentioned. Preserve the original context where it is important for understanding what those counts mean.

Conduct graphing, visual inspection, statistical analysis, qualitative analysis, or other appropriate operations on the data

Statistical procedures can help examine whether outcomes changed, whether groups differed, whether trends emerged over time, and whether particular variables were associated with one another. The type of analysis should match the evaluation design, the quality and amount of available data, and the questions you are trying to answer.

Useful forms of analysis may include:

• Counting, graphing, and visually examining frequencies, rates, or other measures over time
• Looking for meaningful changes in trends, such as clear increases, decreases, or shifts following implementation of a program or policy
• Calculating descriptive statistics, such as the mean, median, mode, percentages, rates, or ranges
• Analyzing interviews, conversations, journals, and participant observations to understand experiences, implementation, and changes over time
• Identifying themes and patterns in qualitative information. When many participants describe similar barriers, supports, experiences, or concerns, those patterns may help explain how the program is functioning and where changes may be needed.
• Comparing results with previously established goals, benchmarks, or implementation expectations

Take note of significant, meaningful, or unexpected results

Findings may be statistically significant, practically important, unexpected, or especially meaningful to participants and communities. These are not necessarily the same thing. A statistically detectable difference may be too small to matter in practice, while an important community-level change may not be captured well by a conventional statistical test.

Types of findings you may want to examine include:

• Changes within individuals, groups, organizations, or communities over time. Repeated measurements can help show whether outcomes changed after an intervention was introduced. Staggered implementation across people, groups, or sites may strengthen confidence that observed changes are associated with the intervention.
• Differences between or among groups. When groups are appropriately comparable, differences in outcomes may provide evidence about program effects. Interpretation should consider how groups were formed, whether they differed at baseline, and the uncertainty around the estimates.
• Statistically significant changes. Statistical tests may indicate that an observed difference is unlikely under a particular null hypothesis. Whether that difference was caused by the intervention depends on the evaluation design and other evidence.
• Correlations and associations. Variables may change together in ways that suggest potentially important relationships. These findings can generate new questions and hypotheses, but should not automatically be interpreted as evidence that one variable causes another.

Correlation does not establish causation. Two variables may change together because one influences the other, because the relationship works in the opposite direction, because both are affected by a third factor, or because the association occurred for another reason.

For example, two outcomes may both increase as young people get older, even though neither causes the other. On the other hand, a strong and repeated association between a medication and an adverse health outcome could provide an important signal that deserves additional investigation. The appropriate response is to examine the relationship carefully rather than assume a causal explanation from correlation alone.

• Patterns. Both quantitative and qualitative data may reveal patterns by geography, access to resources, language, age, race, ethnicity, disability, income, program location, or other relevant characteristics. Such patterns should be explored carefully rather than interpreted as inherent characteristics of a group. They may reflect differences in exposure, opportunity, access, policy, environment, discrimination, or other contextual factors.
• Findings with clear practical importance. Some results may stand out even before advanced analysis. For example, participants in a health program might show meaningful improvements in blood pressure, physical activity, or well-being compared with an appropriate comparison group. Conversely, consistently unchanged outcomes may signal that the program needs adjustment, is not reaching people who could benefit, or is measuring the wrong outcomes.

Important findings do not always answer only the question of whether the program “worked.” Evaluation may reveal that a community-wide issue is concentrated in particular neighborhoods, that some groups experience greater barriers to services, or that outcomes differ according to access to resources or environmental conditions. Examining findings from multiple perspectives can help organizations improve their programs while also identifying broader conditions that may require attention.

Interpret the results

Once you have organized and analyzed the information, the next step is interpreting what it means. A common evaluation question is whether the program contributed to meaningful change. In research terms, this involves examining the relationship between the intervention and the outcomes it was intended to influence while considering other possible explanations.

Several broad patterns of findings are possible:

• The program contributed to the outcomes you hoped to achieve. Evidence may show meaningful improvements associated with implementation, particularly when findings are consistent across multiple measures or stronger than those seen in an appropriate comparison group.
• The program does not appear to have produced the intended outcomes. The evaluation may show little or no meaningful change in the outcomes being measured.
• The program may be associated with harmful or unintended outcomes. These findings require careful interpretation. For example, an increase in reported incidents of interpersonal violence after a program begins could indicate an actual increase, but it could also reflect improved screening, increased awareness, greater trust in reporting systems, or better access to support.
• The program may produce several different effects.
  • Some effects may be beneficial. A youth violence-prevention initiative, for example, might reduce violent incidents while also improving school engagement or participants' sense of safety and belonging.
  • Some effects may be neutral or unrelated to the program's central goals.
  • Some effects may be harmful or unintended. A strategy that improves one outcome might create new barriers, increase disparities, or have unintended effects in another area.
  • Effects may also be mixed, differing across outcomes, participants, settings, or time periods. Understanding who benefits, who does not, and under what conditions is often as important as calculating the average effect.

If the evidence suggests that the program is accomplishing its goals, continue examining how it might be strengthened, sustained, or adapted. Effective programs should continue learning rather than assuming that current results cannot be improved.

Strong community programs are dynamic. They use evaluation findings to continue learning and adapting rather than treating initial success as the end of the improvement process.

If the evaluation shows limited, uneven, or harmful outcomes, interpretation becomes especially important. Is the underlying approach inappropriate, or is a promising approach being implemented inconsistently? Are there organizational barriers, accessibility issues, resource constraints, policy conditions, discrimination, or other structural factors influencing outcomes? Are some groups benefiting while others face barriers? Should particular components be modified, expanded, removed, or redesigned?

Careful interpretation can help answer questions like these. Associations among variables may generate hypotheses about factors contributing to outcomes, but those relationships should be investigated rather than treated automatically as causal. Qualitative information can be especially valuable for understanding why participants experience a program differently and for identifying supports, barriers, or contextual conditions that are not apparent in numerical data.

For example, if participants experience ongoing health challenges, qualitative information might identify barriers related to food access, housing conditions, transportation, affordability of care, language access, previous experiences with health systems, work schedules, caregiving responsibilities, cultural preferences, or the availability of trusted and responsive providers. Interpretation should begin with participants' own experiences rather than assumptions about their knowledge, culture, or choices.

Once your information has been organized and analyzed, both numerical findings and qualitative evidence should be considered together using careful reasoning and critical thinking. Numbers may show that change occurred, while participant experiences, program records, and contextual information may help explain what contributed to it and why.

Analysis and interpretation bring the evaluation process back to where it began: learning how to improve the work. Use what you have learned to adjust the program, continue monitoring outcomes, evaluate those changes, and repeat the process as needed. Ongoing evaluation can help strengthen programs while also identifying organizational, environmental, policy, and systems-level changes that support healthier and more equitable communities.

Understanding an initiative requires attention to context, relationships, culture, history, systems, and participants' lived experiences. In that sense, evaluation often involves learning not only whether something changed, but how people understand and experience that change.

In Summary

The heart of evaluation is collecting meaningful information about a program or intervention and analyzing that information to understand implementation, outcomes, strengths, limitations, and opportunities for improvement.

Quantitative data—information expressed numerically—can be summarized through graphs, descriptive statistics, and more formal statistical analysis. These methods can help identify changes, differences, trends, and associations among variables. Their interpretation should always take the evaluation design, data quality, uncertainty, and practical significance into account.

Qualitative data—including interviews, observations, participant experiences, descriptions of conditions, documents, and other contextual information—can help explain how and why outcomes occurred, what participants experienced, what barriers or supports were present, and what may need to change.

Once you have learned from the information you collected, use those findings to strengthen the next cycle of planning, implementation, and evaluation. Continued data collection, reflection, and analysis can help keep community programs responsive, effective, equitable, and accountable to the people they are intended to serve.

What do we mean by collecting data?

___Collecting data means putting your design for a measurement system into operation

___Collecting data involves gathering information through observation, interviews, testing, surveys, and/or other means; recording it in appropriate ways; and organizing it so that it’s easier to work with

What do we mean by analyzing data?

___Analyzing data involves examining the information you’ve collected in ways that reveal the relationships, patterns, trends, etc. that can be found within it

___Data can be quantitative (collected as numbers) or qualitative (collected as narrative information, records, journal notes, etc.)

___Quantitative data is usually analyzed by subjecting it to one or more graphical displays or statistical operations that demonstrate the significance of relationships among variables

___Data are also usually analyzed logically, by looking for patterns and relationships revealed within them

___Qualitative data can sometimes be turned into quantitative data by, for instance, counting (e.g., the number of times a behavior occurs in various circumstances) or by rating on a number scale such dimensions as importance, satisfaction, or quality (e.g., the quality of housing or quality of life in neighborhoods)

___A combination of quantitative and qualitative data often yields the best overall picture

Why should you collect and analyze data for your evaluation?

___This can show whether or not there was actually any significant change in the dependent variable(s) you hoped to influence

___This can show connections between or among various factors that may have an effect on the results of your evaluation

___This can imply or show the reasons that your work was effective or ineffective

___This can provide you with credible evidence to show funders and the community that your program is successful, or that you’ve uncovered, and are fixing, the elements that are barriers to success

___This can show that you’re serious about evaluation and about improving your work

___This can show the field that what you’re doing works well, and thus pave the way for others to use similar methods and approaches as best practices

When and by whom should data be collected and analyzed?

___Data collection should start no later than when you begin your work and continue throughout

___If you want to see if any change is part of a long-term trend, you should collect data for some time before your program actually starts

___If you want to understand long-term effects, you should collect data on participants for some time after they leave your program

___Data should be collected and analyzed by people who are capable of doing so

___Data collection and analysis can be done by anyone from community members who have been trained to professionals with experience in conducting studies

How do you collect and analyze data?

___Implement the observational system you’ve planned

___Organize the data you’ve collected

___Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

___Take note of any significant or interesting results

___Interpret the results

Example #1: Santa Monica’s Use of Data to Lift All Boats: A Deeper Look

Four years ago, Santa Monica took a data-driven approach to become a true City of Wellbeing where everyone thrives. The city used a Wellbeing Index to produce a data-based guide for steering policy. The city used existing information from city departments and fresh input from surveys. The Wellbeing index sets a framework for priorities and promotes initiatives on health and wellness.

Contributed by Lia Thompson, University of Kansas, Community Tool Box Intern.

 

 

• What is ethnographic information?

• Why might ethnographic information be important to evaluation?

• When might you want to collect ethnographic information?

• Who should collect and interpret ethnographic information?

• How do you gather ethnographic information?

• How do you interpret ethnographic information?

Someone watching a baseball game or cricket match for the first time may be puzzled by a series of actions that experienced players and fans immediately recognize. A batter might adjust equipment, step out of position, look toward a coach, or repeat a familiar routine before each pitch. Some actions may help the player focus, some may relate to strategy or communication, and others may simply be personal habits. Without understanding the context, an outside observer could easily misinterpret what is happening.

Understanding why people act as they do can be important when trying to understand a community, organization, or social group. Ethnographic researchers often spend extended time in the settings they are studying, listening, observing, participating when appropriate, and learning how people themselves understand their experiences. This approach can help reveal not only what happens in everyday life, but also the meanings, relationships, histories, values, institutions, and assumptions that shape people's actions.

Depending on the issue you are evaluating, looking only at individual behavior may provide an incomplete picture. Social exclusion, violence, health inequities, institutional trust, access to services, and many other community issues are shaped by relationships, policies, history, economic conditions, social expectations, and other contextual factors. These cannot always be understood through tests, surveys, or counts alone. Ethnographic methods can help evaluators understand how people experience these conditions and what those experiences mean in the context of their everyday lives.

What is ethnographic information?

Ethnographic information is qualitative information gathered through sustained engagement with members of a community, organization, or social group, with particular attention to how people themselves understand their experiences and surroundings. Its purpose is to describe and interpret social practices, relationships, values, expectations, and meanings from the perspectives of the people involved rather than imposing assumptions from outside.

Depending on the evaluation, ethnographic inquiry may examine community norms, health and well-being, systems of power, religion and spirituality, economic conditions, relationships, identity, institutions, and broader ways of understanding the world.

The terms “culture,” “group,” and “community” are used broadly in this section. A group may be connected by geography, language, religion, ethnicity, profession, school, organization, shared experience, interest, or another form of affiliation. No community or cultural group is completely homogeneous. People within the same group may differ substantially by age, gender, socioeconomic position, disability, immigration experience, political perspective, religion, generation, neighborhood, and many other characteristics. Ethnographic research should therefore look for both shared patterns and important differences rather than assuming that all members of a group think or behave alike.

Community norms. These include expectations and practices that influence everyday behavior and relationships. Areas to explore may include:

• Gender expectations and roles. What expectations exist around gender, caregiving, employment, leadership, family responsibilities, dress, communication, and relationships? How consistent are these expectations across generations and individuals, and how are they changing?
• Social structure and status. How are influence, respect, opportunity, and social status distributed? What institutions or relationships shape access to resources and decision-making? Who is more likely to be included or excluded, and how easily can people move between social or economic positions?
• Behavior and interaction. What forms of communication and behavior are considered respectful, appropriate, or inappropriate in different settings? How do expectations vary according to age, role, relationship, setting, or other factors? What formal or informal consequences exist when community expectations are violated?

Social norms can sometimes support health, safety, belonging, and cooperation, while other norms may contribute to harm, exclusion, discrimination, or unequal treatment. Evaluators should avoid assuming that harmful practices are inherent to a particular culture. Instead, examine how history, policy, institutions, social conditions, and community perspectives interact to shape what is considered normal or acceptable.

Understanding these dynamics can help explain how people perceive a program or initiative and what changes may be needed to make the work more relevant, respectful, and effective.

Health and well-being. Ethnographic inquiry can help explain how people understand health, what resources they use, what barriers they encounter, and what practices support well-being. Areas to explore might include:

• Experiences and practices related to prenatal, infant, child, and family health
• Community experiences with alcohol, tobacco, other substances, substance use disorders, prevention, treatment, and recovery
• Awareness and experience of chronic health conditions, environmental risks, and protective factors
• Access to nutritious food, opportunities for physical activity, and the economic, environmental, and social conditions that influence those opportunities
• Access to and experiences with vaccinations, screenings, preventive care, and other health services
• Relationships with health care providers and levels of trust in health systems
• Access to affordable, culturally responsive, accessible, and timely health care

In communities with limited infrastructure or health resources, relevant questions might include: Do people have reliable access to safe drinking water, sanitation, medications, transportation, and health services? What barriers affect access to care? What local knowledge, practices, community health workers, mutual-aid networks, or other resources already support health and well-being?

Power and political realities. Who has formal and informal influence? Who participates in decisions, and whose perspectives are often excluded? How are laws, policies, and institutional practices experienced by different groups? Which organizations, leaders, networks, or community members are trusted? How do people advocate for change? These questions can help reveal the distribution of decision-making power and the ways policies and institutions affect daily life.

Religion and spirituality. Religion and spirituality may play a central role in some communities and a limited role in others. Relevant questions might include:

• What religious, spiritual, or nonreligious traditions are represented in the community?
• What roles do faith-based organizations or spiritual leaders play in social support, community life, advocacy, education, health, or civic engagement?
• How do community institutions respond to religious diversity and to people with no religious affiliation?
• Which values, traditions, practices, or ethical principles are especially important to community members?
• How do religious or spiritual beliefs influence people's decisions, relationships, health practices, or participation in community life?
• How do religious institutions interact with public institutions, government, schools, health systems, and other community organizations?

Economics. What economic conditions shape everyday life in the community?

• What are local patterns of employment, unemployment, underemployment, wages, and job security?
• Is the local economy heavily dependent on particular employers, industries, occupations, or seasonal work?
• Are there inequities or discrimination in hiring, pay, promotion, or access to economic opportunity?
• How accessible are education, training, transportation, childcare, technology, and other resources connected to economic opportunity?
• What kinds of work, caregiving, informal labor, and other contributions are valued or overlooked?
• What opportunities and barriers affect economic mobility?
• Is safe, stable, accessible, and affordable housing available?
• To what extent do households experience economic hardship, housing instability, food insecurity, or difficulty meeting basic needs?

What constitutes a basic need can vary according to geography, infrastructure, climate, household circumstances, local costs, and social conditions. Rather than assuming a universal standard based on one society's expectations, ask community members what resources they consider necessary for health, safety, dignity, participation, and well-being, and examine whether people have equitable access to those resources.

• How are income, wealth, resources, and economic opportunities distributed, and which groups experience the greatest barriers or disparities?

Worldview and shared understandings. This overlaps with many of the categories above and refers to the ways people understand themselves, their communities, institutions, relationships, and possibilities for change. Relevant dimensions may include:

• Views of authority and institutions. Which people or institutions are considered trustworthy or legitimate, and why? How have historical and current experiences with government, schools, law enforcement, employers, health systems, religious institutions, and community organizations influenced those views? People may trust different sources of authority depending on relationships, experience, accountability, expertise, and whether institutions are seen as acting fairly and in the community's interests.
• Sense of agency and influence. To what extent do people feel able to influence decisions affecting their lives and communities? What strengthens or limits that sense of agency? How do institutional practices, economic conditions, discrimination, legal status, age, access to resources, and community relationships shape people's opportunities to participate in change?

People may locate decision-making power in themselves, their families, community networks, employers, government institutions, legal systems, religious traditions, civic organizations, or other sources. These perceptions often reflect both personal experience and broader social conditions, and they may vary considerably within the same community.

• Relationships and trust. How is trust built and maintained? Which relationships provide support, belonging, caregiving, information, or resources? How are family relationships, friendships, partnerships, intergenerational relationships, and community networks understood? Where do conflict, exclusion, or unequal power appear?
• Responsibilities and rights. What responsibilities do people feel toward family, neighbors, community organizations, or the broader society? What rights and opportunities do people expect to have? Do community members experience those rights equally in practice?

Expectations about mutual support, individual responsibility, collective responsibility, and civic participation differ both within and across communities. These differences should be explored through people's experiences rather than inferred from whether a community is rural or urban, wealthy or economically constrained, or located in a particular part of the world.

Likewise, legal rights may exist formally while access to those rights differs in practice. Ethnographic inquiry can help reveal how people experience voting, property, employment, education, health care, public services, legal protections, and other rights in their everyday lives.

• Inclusion and belonging. Who is welcomed, included, or represented in community life? Who experiences exclusion, stigma, discrimination, or barriers to participation? How do institutions and informal community practices respond to differences in race, ethnicity, disability, language, gender, sexual orientation, immigration experience, socioeconomic position, religion, age, or other characteristics?
• Identity and affiliation. How do people describe themselves, and which relationships or communities are especially important to their sense of identity and belonging? These might include family, neighborhood, cultural or ethnic identity, profession, faith, generation, peer networks, civic organizations, sports, or other communities.

You may not need to study every aspect of a community. In most evaluations, ethnographic inquiry focuses on the specific cultural, institutional, or social factors most relevant to the issue. For example, an evaluation related to school engagement might explore how students experience school, what supports or barriers affect attendance and belonging, how families view the school system, and how institutional practices influence participation.

Several key principles characterize ethnographic information:

• Ethnographic information comes primarily from people who participate in or directly experience the community or setting being studied. Depending on the evaluation, this might include program participants, residents, students, workers, caregivers, community leaders, people from particular age groups, language communities, or others whose experiences are relevant.
• Ethnographic information is generally gathered in natural settings where people normally live, work, learn, gather, or participate. This helps evaluators understand experiences beyond the formal program environment.
• Ethnographic inquiry seeks to understand experiences from the perspectives of the people involved.
• Ethnographic description should aim to understand before judging. Researchers should document perspectives, practices, relationships, and context accurately while remaining attentive to power, harm, inequality, and the researcher's own assumptions. The goal is not to treat every practice as acceptable, but to understand the meanings and conditions surrounding it well enough to support thoughtful and effective action.

Why might ethnographic information be important to evaluation?

Ethnographic information can help explain why programs work differently across people, settings, and circumstances in ways that numerical information alone may not.

For example, ethnographic research has been used to understand why some households and communities are more likely to be missed in population counts. Interviews and observation may identify problems such as inaccurate address records, complex or changing household arrangements, concerns about how information will be used, limited language access, accessibility barriers, distrust based on previous experiences with institutions, or difficulty understanding forms and procedures.

Findings like these can lead agencies to improve outreach, simplify materials, provide information in multiple languages and accessible formats, work with trusted community organizations, and revise data-collection procedures so that more people are represented accurately.

Ethnographic information provides insight into how participants actually experience programs and initiatives. Staff may believe that a service is accessible and welcoming while participants experience long waits, confusing procedures, transportation barriers, disrespectful treatment, language barriers, or requirements that conflict with work and caregiving responsibilities. Ethnographic inquiry can reveal these differences.

Ethnographic information can help identify what should change for an approach to be more useful and responsive. Understanding participants' experiences can help organizations adjust communication, scheduling, staffing, services, policies, physical spaces, or other aspects of the program.

Ethnographic information can clarify how people understand an issue and how important it is in their lives. Community members may define an issue differently from program staff or researchers. Understanding those perspectives can improve how problems and goals are framed.

Ethnographic information can deepen your understanding of community context. This can strengthen future planning, relationships, advocacy, implementation, and evaluation while reducing the risk that programs rely on stereotypes or assumptions.

When might you want to collect ethnographic information?

Ethnographic research can provide valuable insight, but it also requires time, trust, skill, and sustained engagement. You may not need an extensive ethnographic study for every evaluation.

• When you are working with a community or social group whose experiences and context are unfamiliar to your organization. Ethnographic inquiry can help challenge assumptions and identify issues that outside observers might otherwise overlook.
• When you are working with a group that has developed shared norms or practices. A school classroom, faith community, workplace, neighborhood organization, sports team, peer network, or other group may develop its own expectations and relationships that influence how a program is experienced.
• When understanding community context is fundamental to the issue. Issues such as racism, discrimination, violence, institutional trust, health inequities, or social exclusion cannot be fully understood without examining history, relationships, policies, power, opportunities, and people's lived experiences.
• When your evaluation focuses on a particular population, setting, or experience. Understanding how people experience a job-training program, health initiative, school program, public service, or other intervention can help explain what is working, what creates barriers, and why.

Who should collect ethnographic information?

Who collects ethnographic information depends on the purpose of the evaluation, available time and resources, existing community relationships, and the level of trust required. Trained evaluators or researchers may conduct the work, but community members, peer researchers, program participants, or other people with relevant lived experience can also play central roles.

In many situations, involving community members as research partners can improve trust, interpretation, cultural responsiveness, and access to perspectives that might otherwise be missed. Community researchers should receive appropriate training, compensation, support, and decision-making authority rather than being used only as sources of access or information.

How do you gather ethnographic information?

Begin by deciding what you need to understand. Then identify appropriate methods, build relationships and trust, develop an ethical fieldwork plan, and carry out the research in collaboration with the people and communities involved.

1. Decide what kinds of information you need. You may be interested in a broad understanding of a community or in a specific question connected to your evaluation. Most evaluation-focused ethnographic research examines particular aspects of culture, institutions, relationships, or daily experience rather than attempting to describe an entire culture.

Define the topic clearly enough to guide observation and conversation. Useful questions might include: How do participants experience the program? Which parts feel relevant or useful? What creates barriers? Are organizational goals aligned with participants' goals? How do staff-participant relationships influence participation? Are language, accessibility, scheduling, institutional practices, or prior experiences affecting engagement?

Suppose, for example, you want to understand how students experience an English-language learning program. The focus would be the learning environment and students' experiences rather than assumptions about any particular ethnic or national culture.

Questions might include:

• What goals do students have for learning English, and how does the program support those goals?
• How do students experience relationships with instructors and other students?
• Which teaching and learning approaches help students participate, practice, and build confidence?
• What supports or barriers outside the classroom—such as work schedules, transportation, childcare, technology, or prior educational experiences—affect participation?
• Which topics, examples, learning activities, and teaching approaches do students find relevant and useful?
• What do students discuss among themselves about the class and their learning experiences?
• Are there language, identity, power, accessibility, or interpersonal dynamics that affect participation and belonging?

Findings might suggest changes to class schedules, instructional materials, language supports, teaching methods, group activities, staffing, technology access, childcare support, or other aspects of the learning environment. Adjustments should be based on what students report and experience rather than assumptions about their cultural backgrounds.

Knowing which questions you want to answer will help determine whom to engage, what settings to observe, how much time will be needed, and which methods are appropriate.

2. Determine what you have the resources to do. An evaluation-focused ethnographic study does not need to involve years of immersion. A community organization may gain more useful information by spending sustained time in participants' neighborhoods, workplaces, schools, community spaces, or program settings than by attempting a broad study of an entire national or ethnic culture.

3. Build trust and relationships with the community. Trust is not simply a means of obtaining information; it is an ethical foundation of community-based research. Explain who you are, why the research is being conducted, what will happen with the information, and how community members can influence the process.

Organizations with established community relationships may already have a foundation of trust. When that is not the case, working with trusted community organizations, peer researchers, neighborhood leaders, faith leaders, cultural organizations, advocates, or other respected community members may help establish relationships.

Avoid treating any individual as a spokesperson for an entire community. Trusted partners can help with introductions and context, but the research should still include a range of perspectives.

When engaging people in community settings, choose spaces and methods that are appropriate, accessible, and comfortable for participants. Respect people's privacy and avoid approaching individuals in ways that feel intrusive or coercive.

4. Plan your field study. Ethnographic fieldwork takes place in the settings where people normally live, work, learn, gather, or participate. Depending on the evaluation, this might include classrooms, community centers, workplaces, public spaces, meetings, homes when invited, online communities, or other relevant environments.

By this point, you have already completed some of the planning. Additional steps include:

• Decide on your questions. Ethnographic questions should help explain participants' experiences, relationships, and context in ways that inform the broader evaluation.
• Determine which methods you will use. Possible approaches include:
  • Individual or group interviews, ranging from informal conversation to structured or semi-structured interviews
  • Participant or non-participant observation
  • Informal conversations and relationship-building
  • Participation in community activities, meetings, celebrations, work, or other relevant settings when invited and appropriate
  • Open-ended surveys or questionnaires
  • Interviews with people who have relevant knowledge or experience
  • Documents and cultural materials, such as organizational communications, policies, websites, social media, artwork, meeting records, educational materials, media, or other sources relevant to the group
  • Demographic, administrative, and public data used carefully alongside community perspectives
  • For more detail on several of these approaches, see Chapter 3, Section 15, Qualitative Methods to Assess Community Issues.
• Decide whom you need to engage and how. Seek a range of perspectives rather than relying only on formal leaders or the people easiest to reach. Recruitment methods should respect privacy, voluntary participation, language access, accessibility, and people's right to decline.

Ethical considerations: Your first responsibility is to the people and communities participating in the research. Explain the purpose of the study, what participation involves, how information will be recorded and used, possible risks and benefits, and how privacy and confidentiality will be protected. Obtain appropriate informed consent and make clear that participation is voluntary.

Consider who owns or controls the information collected, who will have access to it, how long it will be retained, whether participants or community partners will be able to review findings, and how results will be shared back with the community. Researchers should avoid extractive practices in which communities provide knowledge but have little influence over interpretation or use of the findings.

Ethnographic research also requires reflexivity—ongoing attention to how researchers' identities, relationships, assumptions, institutional roles, and power can influence what they observe and how they interpret it. Accuracy does not come from pretending that researchers have no perspective, but from documenting and critically examining how that perspective may affect the work.

5. Carry out your fieldwork and keep fieldnotes. Fieldwork means spending time in relevant settings, developing relationships, observing, listening, participating appropriately, and gathering information using the methods you planned.

Your methods may change as you learn more. Ethnographic inquiry is often iterative: early observations lead to new questions, which guide later conversations and observations. Document those changes and why you made them.

Fieldnotes are records of observations, conversations, context, questions, interpretations, and researcher reflections. They should be written during or as soon as practical after fieldwork while details are still fresh. Audio, photographs, or video may also be useful when appropriate, but only with clear permission and suitable privacy protections.

Useful elements of fieldnotes may include:

• Date, time, setting, and relevant context
• Specific observations and details about what occurred
• Relevant sensory and environmental observations
• Reflections on how the researcher's presence may have influenced the setting
• Important words, phrases, terminology, and summaries of conversations
• Questions or issues for future investigation
• A consistent system for organizing notes and linking them to dates, settings, participants, or themes

Fieldnotes can include several distinct types of material:

• Jottings are brief notes made during fieldwork to help the researcher remember details that can be expanded later.
• Description records what happened, who was involved, relevant context, and other details of the setting or experience.
• Preliminary analysis records emerging patterns, questions, interpretations, and possible connections while making clear that these ideas may change as more information is gathered.
• Researcher reflection documents reactions, assumptions, uncertainties, relationships, comfort or discomfort, and ways the researcher's position may influence what they observe or how others respond.

Reflexive notes can help evaluators identify assumptions and recognize when a program may conflict with participants' priorities or experiences. The goal is not to label community perspectives as incorrect or deficient, but to understand where differences exist and how programs can respond respectfully while still addressing safety, equity, or other important goals.

Timely, detailed, accurate, and ethically maintained fieldnotes are central to strong ethnographic research.

How do you interpret ethnographic information?

Ethnographic information becomes useful when it helps explain the relationships, experiences, conditions, and meanings connected to your program and evaluation. Begin by organizing fieldnotes and other materials systematically.

Organize the final version of your notes. You might categorize information by theme, setting, time period, type of interaction, participant role, or other dimensions relevant to your questions. Demographic characteristics may also be useful when they are relevant, but avoid assuming that differences are caused by identity alone. Patterns may instead reflect differences in access, treatment, opportunity, exposure, institutional practices, or other contextual factors.

Develop a description of the setting or community from participants' perspectives. Depending on the evaluation, this may be broad or narrowly focused on a particular program, service, policy, or experience.

Look for connections among relationships, institutions, history, resources, identities, practices, and other contextual factors. The goal is not to reduce a community to a single “culture,” but to understand how different factors shape people's experiences and participation.

Consider an English-language learning program in which some students participate actively in certain activities but speak less in others. Rather than assuming that their behavior reflects a fixed cultural trait, the instructor could explore multiple explanations: comfort with the language, previous educational experiences, confidence, classroom power dynamics, fear of making mistakes, teaching methods, relationships among students, accessibility, or expectations about participation.

Students themselves might suggest solutions such as smaller discussion groups, additional practice time, peer support, different participation formats, more relevant lesson topics, or opportunities to contribute without being called on unexpectedly. Ethnographic inquiry helps the instructor understand the actual reasons behind participation patterns before deciding how to respond.

Reexamine your initial interpretations. What did you learn about participants' experiences, expectations, relationships, and context? Which early assumptions were supported, challenged, or complicated by later information? Revise your interpretations as your understanding develops.

Try to understand your original question from the perspectives of the people involved.

• Do community members define the issue in the same way you do?
• Are there contextual factors you were previously unaware of?
• Are participants interpreting program activities or conditions differently from what you expected?
• How important is the issue to participants, and what other priorities compete for their time and attention?
• Do participants and the organization share the same goals? If so, do they agree on how those goals should be pursued?
• Are program practices, institutional expectations, or community conditions creating barriers to participation or success?
• What community strengths, relationships, practices, or resources support participation and positive outcomes?
• Are there unintended or unexpected effects related to context, identity, relationships, or institutional practices?

In a language-learning class, for example, participation patterns might be influenced by differences in language proficiency, prior educational experience, age, confidence, classroom relationships, learning preferences, responsibilities outside class, or expectations about speaking in groups. Rather than assuming the reason, ask students and observe how these factors interact.

Translate what you have learned into information that helps answer your evaluation questions. Consider:

• How do community context, relationships, institutions, norms, and lived experiences influence program effectiveness?
• Are program methods culturally responsive, accessible, and appropriate to participants' goals and circumstances?
• Are any activities, policies, or requirements unintentionally creating barriers?
• What changes could make participation more accessible, respectful, useful, and rewarding?
• Which parts of the program appear most useful, and what does the ethnographic information suggest about why?

Continue gathering ethnographic information when it remains useful. Communities, institutions, programs, and relationships change over time. Continued listening and engagement can help organizations recognize those changes and adapt their work accordingly.

In Summary

Ethnography is an approach to understanding people's experiences, practices, relationships, and social context from their own perspectives. In evaluation, ethnographic information can help explain why participants experience programs differently, why particular strategies work well or create barriers, and how history, institutions, culture, relationships, and community conditions influence outcomes.

The group or setting being studied might be a classroom, neighborhood, workplace, organization, faith community, professional network, or broader social or cultural community. Whatever the setting, ethnographic inquiry requires time, relationship-building, careful observation, listening, ethical practice, and continued attention to diversity within groups.

Planning an ethnographic study involves identifying what you need to understand, determining what resources are available, building trust, choosing appropriate research questions and methods, engaging a range of participants, and documenting observations and conversations carefully. Ethical practice requires informed consent, privacy protection, transparency, attention to power, and meaningful community involvement in how information is interpreted and used.

Organizing and interpreting ethnographic information can deepen your understanding of evaluation findings and help you improve programs in ways that are more responsive to the people and communities involved. Continued engagement can help ensure that your understanding evolves as community priorities, relationships, and conditions change.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

What is ethnographic information?

__ Ethnographic information is information about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view.

__ Elements of ethnographic information include:

• Community norms
• Health conditions and knowledge
• Power and political realities
• Religion
• Economics
• World view

__ Ethnographic information is obtained directly from those who live it.

__ The gathering of ethnographic information takes place in the environment where members of the culture normally spend their time (natural setting).

__ Ethnographic information is meant to help you understand a culture from the point of view of its members.

­­__ Ethnographic information is simply descriptive, not judgmental.

Why might ethnographic information be important to evaluation?

__ In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

__ Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them.

__ Ethnographic information helps to clarify what needs to be addressed in order for participants to respond to your approach.

__ Ethnographic information can clarify the issue and its effect on and importance to the population of interest.

__ Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future.

When might you want to collect ethnographic information?

__ When you’re engaged with a population or cultural group that you’re not familiar with or part of.

__ When you’re working with a clearly-defined group that has had a chance to develop its own culture.

__ When an understanding of the context and culture of the community is fundamental to what you’re doing.

__ When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group.

Who should collect and interpret ethnographic information?

__ That depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc.

How do you gather ethnographic information?

__ Decide what kinds of information you need.

__ Determine what you have the resources to do.

__ Gain the trust of the group you’re concerned with

__ Plan your field study

• Decide on your questions.
• Decide on your methods of data collection.
• Decide whom you’ll need to contact and how.
• Work out ethical issues.

__ Carry out your field study, taking careful field notes, including jottings, descriptions, analyses, and personal reflections on what you learned.

How do you interpret ethnographic information?

__ Organize your data.

__ Write as complete a description as possible of the culture or the element(s) of the culture you’re concerned with from the perspective of its members.

__ Reexamine the analyses from your field notes.

__ Try to understand the answer to your original question from the point of view of those you’re concerned with.

__ Translate what you’ve learned about participants into answers to, or supporting information for answers to, your evaluation questions.

__ Continue to gather ethnographic information to guide your work.

Existing data can help you evaluate whether an intervention is working without requiring you to collect every piece of information yourself. When relevant, reliable data are already available, using them can save time and resources while strengthening your evaluation. It is important to have data to evaluate the effectiveness of an intervention.

To support an evaluation, existing data must be relevant to your population, issue, intervention, and evaluation questions. They also need to be available in a form you can analyze. You data may need to clean, combine, or otherwise prepare the data before use. This section explains how to find, assess, and use existing data thoughtfully.

WHAT IS EXISTING DATA?

Existing data is information that an organization, agency, researcher, or other source has already collected for reporting, program administration, research, or another purpose. Examples include administrative and program records, public datasets and dashboards, surveys, research studies, and community assessments. Some sources are updated regularly, while others provide a snapshot from a particular point in time.

 

COMMON SOURCES OF EXISTING DATA 

Some of the most common sources of data that may be useful for your evaluation include: 

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry 

Existing data is often available through online dashboards, open-data portals, reports, electronic record systems, downloadable files, or direct requests to the organization from which the data originated. Some information will still require a formal data request or data-sharing agreement, particularly when it contains confidential or sensitive information. 

TYPES OF EXISTING DATA YOU MIGHT LOOK FOR 

Depending on your evaluation questions, you might look for data about a specific population or geographic area, including: 

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population 

WHY COLLECT AND USE EXISTING DATA?  

There are sometimes good reasons to collect new data, including when the information you need is not available elsewhere. When you collect new data, you also have more control over who is included, what is measured, and how the information is gathered. However, primary data collection can be time and resource intensive. 

If the information you need, or something very close to it, already exists, there are several good reasons to find and use it. 

• Existing data can save substantial time and effort, especially when you need information about a large population, geographic area, or time period. They may also provide broader coverage or greater detail than your organization could collect on its own.
• Existing datasets may already have been cleaned, organized, analyzed, or documented by people with specialized expertise. This work can reduce the technical burden on your team, although you should still review the methods, definitions, and limitations before using the results.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The existing data you find may be more sweeping or more specific than what you’d be able to gather. They may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Existing data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for.  In cases like these, the use of data that already exist might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• The data may have been collected using established methods and systems. This can save time and strengthen your evaluation, but it does not guarantee that the data are free from bias, missing information, inconsistent definitions, or other limitations. You still need to understand how the data were collected.
• Existing data can help you examine change over time. Historical information can show whether an observed change is part of a longer trend and whether similar patterns appear in comparable communities, the state, or the nation. This perspective may be difficult to obtain through short-term data collection alone.
• Existing data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exist elsewhere. 

WHEN SHOULD YOU COLLECT AND USE EXISTING DATA? 

• When it’s available and accessible. This is the key question. If you know the data exist and can obtain appropriate access, they may be useful. If the data do not exist, are too difficult to locate, or cannot be shared because of privacy or other restrictions, you may need another approach.
• When it’s relevant. As with availability, the relevance of the data to what you’re trying to find out is a key issue. All the existing data in the world won’t do you any good if they don’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, existing data may be the only source of the information you need.
• When the data match your evaluation needs. Data should align with the population, place, time period, measures, and program elements you are studying. Select only sources that can help answer your evaluation questions; otherwise, the information may provide an incomplete or misleading picture. 

HOW DO YOU FIND AND USE EXISTING DATA? 

As you search for and use existing data, there are several questions you should ask. 

Are the data a good fit for your evaluation? 

Before using an existing source, look beyond whether it is easy to obtain. Ask whether it can answer your evaluation question and whether it can be meaningfully compared with any data you collect yourself. 

• Who and what do the data represent? Make sure the population, geographic area, and issue are relevant to your program.
• When were the data collected or updated? Older data can be useful for trends and context, but may not reflect current conditions.
• How were the data collected and defined? Review the data source’s methods, measures, and definitions. Two sources may use the same term—for example, “service received” or “graduation”—but mean different things.
• Are the data complete and sufficiently detailed? Consider missing data, small numbers, the level of detail available, and whether results can be broken down in ways that are useful and appropriate.
• What limitations or potential biases should you keep in mind? Administrative data reflect the people who use a service and the way staff record information; survey and research data may be affected by sampling, response rates, or measurement choices.
• Can you use the data ethically and securely? Follow data-sharing agreements and privacy requirements. Whenever possible, use aggregate or de-identified data rather than information that could identify individuals. 

WHAT INFORMATION DO YOU NEED, AND WHY? 

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants. 

Some possibilities include: 

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• Community-level indicators are specific, measurable signs of conditions or change across a community. Examples include program participation, services delivered, crime rates, or new cases of HIV. These indicators can help you track community trends and assess whether an initiative may be contributing to change. 

Community-level indicators describe trends for the community as a whole, rather than changes among individual participants. For example, an initiative to reduce alcohol use among young people might track weekend and nighttime single-vehicle crashes involving teenagers. Choose indicators because they are meaningful to the initiative and community—not only because data happen to be available. 

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include: 

• Demographics:
  • Demographic characteristics, such as age, race, ethnicity, gender identity, household composition, education, income, employment, housing, disability status, primary language, and geographic location
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level  
  • Other characteristics –primary language spoken, etc. 
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Alcohol and other substance use
  • Sexual behavior  
  • Various other behaviors – work habits, consumer patterns, etc. 
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Developmental and educational outcomes, such as completion of primary or secondary education and equitable participation by people with disabilities
  • General health and well-being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins) 

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes. 
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

Combining several years of data can produce more stable estimates when annual counts are small. Rolling averages—for example, comparing 1997–2000 with 1998–2001—can help show change over time while reducing year-to-year variation. The tradeoff is that multi-year estimates may make recent changes or program effects harder to detect.

Expanding the geographic area can also increase the number of observations and improve statistical stability. For example, you might use regional or state data when county-level counts are too small. However, broader estimates may not reflect local conditions and can hide meaningful differences among communities. 

Who is likely to have collected that information?

Public agencies, research organizations, universities, schools, health and human service providers, funders, advocacy groups, community organizations, and businesses may all maintain useful data. Start with the source most likely to collect information about your topic, population, or geographic area.

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

WHERE SHOULD YOU LOOK FOR EXISTING DATA? 

The question here is not only where to find existing information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble. 

Your own records and data 

Unless it’s brand new, your own organization should have administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information. 

Online data portals and websites 

Many public documents and datasets are available online. Start with the website or open-data portal of the government agency, organization, or institution most likely to have collected the data. In the U.S., federal, state, tribal, county, and municipal agencies often provide searchable reports, dashboards, or downloadable files. Similar sources are available in many other countries.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

Assess online data sources carefully. Favor information from the organization that originally collected or maintains the data, and look for clear documentation of methods, definitions, update schedules, and limitations. If a source is unfamiliar or a finding seems uncertain, compare it with other credible sources before using it. 

Go directly to the source

When data is not publicly available, contact the organization that maintains them. Explain what information you need, how you plan to use it, the level of detail required, and how you will protect privacy. Be prepared to discuss a data-sharing agreement, reasonable costs, or a mutually beneficial exchange of information or expertise. 

When requesting data from another organization for comparison, frame the conversation around shared learning rather than competition. Explain why the comparison is useful, how you will account for differences between programs or populations, and how findings will be interpreted and shared. An existing relationship built on trust and mutual respect can make this collaboration easier. 

Libraries

Libraries and librarians can help you identify credible datasets, government reports, scholarly research, archives, and specialized databases. They may also help you develop search terms, locate hard-to-find sources, and assess whether a source is authoritative. 

What are you planning to do with the data once you have it?

Plan how you will use the data before requesting or downloading them. Determine which variables, time periods, population groups, and level of detail you need. If you intend to compare existing data with information from your program, make sure the groups, definitions, measures, and collection periods are sufficiently similar to support a meaningful comparison.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the existing data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending. 

USING EXISTING DATA 

In many evaluations, you will combine existing data with new information collected from participants. Existing program or administrative records may also provide current information about participants, depending on what is recorded, how quickly the data are updated, and whether you have appropriate access. You may be able to find earlier data on participants to use as a baseline or data on a similar group to use as a comparison or control. 

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find existing data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both. 

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use existing data: 

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. A clearer picture of the community allows for a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved because of your work. 

In an evaluation, the intervention is the program, policy, service, or other action expected to contribute to change. The outcomes are the behaviors, conditions, or other results that may change. For example, a violence-prevention program is the intervention, while changes in violent behavior and related injuries are outcomes. 

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding. 

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group. 

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use existing data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change. The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find existing data that tells you whether the effects of your program continued, kept growing, or trailed off. 

In Summary

Existing data can help you understand community conditions, establish a baseline, identify trends, compare outcomes, and examine change over time. Useful sources include your own records as well as data maintained by public agencies, community organizations, health and human service providers, universities, funders, and other institutions. Select data that fit your evaluation questions, examine their quality and limitations, and follow all privacy and data-sharing requirements. Used thoughtfully, existing data can make evaluation more feasible and informative—especially for organizations with limited time and resources.

What are archival data?

___Archival data is data that already exists as a result of administrative procedures or past studies or evaluations

___Be aware of sources of archival data:

• Public records
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

___Be aware of the possible types of data available:

• Demographics
• Behavior
• Health and development outcomes
• Attitudes – racial, political, social, etc.
• Knowledge and awareness of issues
• Environmental conditions or factors affecting the population and/or your work

Why collect and use archival data?

___It’s easier and less time-consuming than collecting all the data yourself

___Archival data may have already been processed by people with more statistical expertise than you, making it easier to use in analysis

___Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources

___It’s quite possible that you can find more information than you’d be able to gather if you did it yourself

___Archival data could touch on important areas you might not have thought of, or identify patterns or relationships you wouldn’t have looked for

___It may eliminate the need to correct for such problems as lack of inter-rater reliability or observer bias

___Archival data allows the possibility of looking at the effects of your work over time

___Archival data can make it possible for small organizations with limited resources to nonetheless conduct thorough evaluation studies

When should you collect and use archival data?

___When it’s available

___When it’s relevant

___When you don’t have the time and/or resources to collect the data yourself

___When it can truly inform your evaluation

How do you collect and use archival data?

___Determine what information you’re looking for and why

Possible categories of information you’d be interested in:

___Data on past participants

___General information on the population and/or the community you’re working with

___Specific information on appropriate characteristics of the population you’re working with

___Cultural information

___Data on a similar group that can be used as a control or comparison

___Results of previous studies

___Determine who is likely to have collected that information:

• Government departments and agencies, the census, and other entities whose data are likely to be available in public records
• Research organizations
• Academia
• News media
• Foundations and other private funders
• Hospitals and other health care providers
• Mental health providers
• Human service and other community-based organizations
• Advocates and watchdog organizations
• Community activists
• Community economic development organizations
• Businesses and corporations

___Decide where you should look for archival data

• Your own archives
• The Internet
• The original source
• Libraries

___Decide what you plan to do with the data once you have it

Use archival data:

___To better understand the context of your evaluation

___To identify areas to address

___To establish a baseline against which to measure your results

___To identify already-existing trends that may affect the results of your evaluation study

___To establish a standard of comparison against which to measure your efforts

___To act as a control or comparison group

___To provide data for a longitudinal study

Tool 1: Common Secondary Data Sources Used to Address Gender-Based Violence

This tool from Strategic Prevention Solutions provides descriptions of common secondary data sources useful for addressing gender-based violence, along with common challenges with secondary sources.