Section 2. Information Gathering and Synthesis

Learn how to gather information about your community issue and put that information together to design an evaluation to address your questions.

 

• What do we mean by information gathering and synthesis?

• Why gather and synthesize information?

• When should you gather and synthesize information?

• Who should gather and synthesize information?

• How do you gather and synthesize information?

Suppose you wanted to design a house that used very little energy, took few resources to build and maintain, and was affordable for most families. You might have some original ideas about how this could be done, but you’d want to find out what ideas others had as well. You’d probably read about earth-bermed houses (houses that are built into a hillside or earth mound), solar panels or windmills for producing electricity, efficient insulating windows, waste-water recycling, and non-toxic building materials that reuse waste wood and metal. You’d talk to people who built or owned energy-efficient houses, to hear about the realities of living green. You’d learn about the barriers to some environmentally-friendly strategies, as well as ways to get around those barriers. There’s a huge amount of information out there, and it would make sense to gather as much of it as possible, so that you could put together the information, incorporate appropriate elements into your design, and get new ideas based on what’s already been done.

The same is true if you’re designing an intervention or program to deal with a community health or other issue, or an evaluation of that program. Others have also undoubtedly tried to address that issue, some with success and some without. Knowing what they did, how they did it, and what the results were can help you decide how to design your effort. You might be able to find a method here, and a technique elsewhere that all fit together into exactly the program that will suit the people and conditions in your community. Or you might realize that something you’d intended to do simply hasn’t worked in a number of other instances, and so wouldn’t be likely to work for you, either.

Gathering and using others’ ideas doesn’t mean that you can’t use your own or come up with something new. New ideas tend to come out of what others have attempted. Most artists start out imitating others before they develop their own styles. Einstein didn’t just chance on relativity; he was familiar with it because others had worked on it. You can usually innovate more effectively if you know what’s been tried.

This section looks at gathering all the information you can about your community issue and about attempts to address it, and putting that information together to design an evaluation to address your questions. Although this chapter is about evaluation, much of the material in these sections applies to planning the intervention (or program) and the evaluation: the two really can’t be separated.

An evaluation is a research project: we are trying to discover what works and under what conditions.  The steps for designing and using an evaluation – the subject of this chapter – are essentially the same as those for designing the program you’re evaluating.  The elements that you borrow from others’ successful efforts, and those that you create yourself, will give you an intervention and related evaluation questions. Although this section talks about program design, it also applies to the design of the evaluation.

What do we mean by information gathering and synthesis?

Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it. The more information you have about the issue itself and the ways it has been approached, the more likely you are to be able to devise an effective program or intervention of your own.

There are obviously many sources of information, and they vary depending on what you’re looking for. In general, you can consult existing sources or look at “natural examples,” examples of actual programs and interventions that have addressed the issue. We’ll touch on where to find both here, and then go into more detail about them later in the section.

• Existing sources. This term refers to published material of various kinds that might shed light either on the issue or on attempts to deal with it.  These can be conveniently divided into scholarly publications, aimed primarily at researchers and the academic community; mass-market sources, written in a popular style and aimed at the general public; and statistical and demographic information published by various research organizations and government agencies.
• Natural examples. These are programs or interventions developed and tried in communities that have addressed your issue. Studying them can tell you what worked for them and what didn’t, and why. By giving you insight into how issues play out in your or other communities, they can provide nuts-and-bolts ideas about how to (or how not to) conduct a successful program or intervention. For the most part, information sources here are the people who are involved in efforts to address issues similar to yours, or those who can steer you to them. Additionally, there are a number of natural examples (such as single case studies) that have been written about descriptively in the literature of community psychology or public health that may be relevant to your work.

Synthesis is from the Greek; it means putting together. Its English meaning is the same: the putting together of something out of two or more different sources. Synthetic fabrics, for instance, are called that because they’re constructed from a number of different chemical building blocks.

In this section, we’re talking about ideas. Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something new that meets the needs of the community and population you’re working with.

Synthesizing in this way requires identifying the functional elements of each idea or program that you’ve looked at that seems to hold lessons for your work. Functional elements are the core components of each program – the methods, framework, activities, techniques, and other aspects – that make up the specific program you’re examining.  Once you’ve separated these parts out, you can put those that meet your needs together with what you’ve learned about the issue and your own ideas to build a program that speaks specifically to your situation.

As we’ve mentioned, the activities of information gathering and synthesis are needed both to create the original program and to develop an evaluation of it that will help you maintain and improve it.  The two really start in the same place, with what you think will address the issue – what shape the program or intervention should take, with whom it should be applied, and what behaviors or conditions it aims to change. This also informs what its short- and long-term goals should be, and by what means you’ll try to achieve those goals.  Once these are determined, they in turn determine your evaluation questions. You can’t construct an evaluation without knowing exactly what you’re trying to evaluate.

Why gather and synthesize information?

If you’re in the process of starting a program to address a community issue, such as violence or early childhood education, you probably know quite a bit about that issue already.  You’ve dealt with it, perhaps, in a variety of ways, and you have some pretty good ideas about what kind of program would work.  Why take the time and trouble, for you and for others engaged in a participatory planning effort, to read a lot of material written by others and to track down people who’ve run programs? If you’re inclined to think this way, there are a lot of good reasons why you should think again. Gathering information beforehand and putting together what you’ve learned could be the most important things you do to make your program effective.  Here’s why:

• It will help you avoid reinventing the wheel. A lot of different organizations have likely approached this issue before you.  Some might have been successful and some might not have, but all of them have probably learned something that would be useful to you in the process. You don’t have to make the same mistakes someone else did if you know about them, and you don’t have to make up something from scratch that may or may not work, when you have a model that has worked.

It’s certainly not a bad thing if you have some of the same good ideas that others have had, but it helps to know that they are good ideas. And there’s a chance that you might have some of the same bad ideas others have had, in which case it helps even more to know that they’re bad ideas.  It will save you a huge amount of trouble, and perhaps be the difference between creating a program that does its job well and one that fails miserably and disappears. Square wheels don’t roll – someone could have told you that.

• It will help you to gain a deep understanding of the issue so that you can address it properly. The first step in figuring out how to deal with an issue is to know what you’re dealing with. The better you understand it – its causes, how it occurs, how people react when they’re affected by it, what its consequences are for individuals and the community, and who can influence it – the more likely it is that you’ll be able to determine how to approach it.
• You need all the tools possible to create the best program you can. Foremost among the tools you need to plan and implement a program or intervention are information, information, and information. Just as with the issue itself, the more you know about what works for whom, how to make things happen, and how to establish or eliminate certain conditions, the more likely that you’ll be able to plan a successful program that addresses all aspects of the issue and leaves nothing to chance. Various kinds of professional and interpersonal skills may help you implement a program, but if what you’re implementing isn’t effective, it doesn’t matter how skillfully you carry it out.
• It’s likely that most solutions aren’t one size fits all. The more information you gather, the greater the variety of approaches, methods, and frameworks you’ll have to choose from. Putting together the right combination will help you to successfully address the particular needs of your community and population.
• It can help you to be culturally sensitive. Not only can you learn more about the culture(s) of the people you’re working with, but you can probably find a number of approaches that have worked with the cultural group you hope will benefit. Perhaps even more important, you can learn to avoid costly mistakes that may take a lot of time and effort – or be impossible – to repair.
• Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program. As we discussed at the beginning of this section, new ideas seldom spring from nowhere.  They’re stimulated by your own experience and the ideas and experience – both good and bad, positive and negative – of others. Look to the experience of other fields, communities, and countries. The more different ideas you’re exposed to, and the more ways you can put them together, the greater chance there is that you’ll come up with something new that’s more effective than what’s gone before.

When should you gather and synthesize information?

Information gathering and synthesis is crucial to the success of the program and to the relevance and effectiveness of the evaluation. It should start at the beginning of any effort, and contribute to the initial planning. It should also go on throughout the life of the program, so that you can continue to adjust by adding or changing program elements to enhance outcomes, and to generate new ideas.

Major adjustments should generally come at the end of an evaluation cycle, when you have solid information about what worked and what didn’t. That doesn’t mean that you can’t make smaller adjustments in the course of the program to improve results along the way.

There’s a tension here between continually changing a program to make it better and obtaining accurate evaluation results. If you change a method or activity in midstream, your evaluation will not be able to give you a clear assessment of its effectiveness.

How much changing you do in the course of a program depends on your intent.  If your first responsibility is to find out what works best, so you can pass it on, then it’s important not to make changes until an evaluation has been completed.  If your primary responsibility is to the current participants in the program, then you should make whatever changes are necessary whenever they’re necessary to ensure the best outcome for them.

There can be ethical issues involved here.  In medical experiments with new therapies or drugs, for example, some participants are given the new treatment and others aren’t (all participants consent to this arrangement, and to not knowing which group they’ll be assigned to.)  If the new treatment proves to be harmful, there is an ethical obligation for the researchers to stop administering it.  If, on the other hand, it quickly proves remarkably effective, researchers usually feel ethically bound to extend it to others in the study as soon as they can prove its positive effects.  Not all programs necessarily pose ethical problems that are as clear-cut as those encountered in medical studies, but ethical issues should always be considered.

Who should gather and synthesize information?

The assumption throughout this chapter is that the whole process – planning, design, implementation, and evaluation – involves multiple stakeholders.

Typical stakeholders in a community program or intervention might include:

• Program participants or beneficiaries
• Program staff and administrators
• Others affected by the program – police, medical staff, teachers, etc.
• Academics or other researchers
• Local officials
• Community activists

Information gathering

In a participatory process, information gathering can be enhanced by a division of labor determined by the skills and experience of the participants. If there are academics or other professional researchers involved, it would probably make the most sense for them – or others with research experience – to review the evaluation literature. Members of the affected population might be the best ones to collect information about the history of the issue in the community, and about how it currently affects people. Program directors and staff would probably have the best contacts in the field, and thus the best chance to find information about other similar programs. Those with Internet access and computer experience might be the logical on-line searchers, or might act as technical support for others to help them find what they’re looking for. Those with knowledge in the law and legislation might be the ones to examine policies.

There’s also the possibility that training could be provided to the whole group, or to various individuals to allow them to pursue various lines of inquiry. There’s no reason, for instance, that people without research experience couldn’t learn to understand and interpret demographic information or contact programs in other places. (There are some limitations here: levels of related education, materials or computers, and/or inability to connect with other people might all figure in to what kind of research it makes sense to ask others to do.)

Synthesis

It is especially important that all participants in the process be involved in putting together the information. Training new participants to synthesize information will pay dividends in the end, because they may be able to see things in the information that aren’t obvious to experienced researchers. They may know things about the community that shed light on which elements of other programs might be appropriate and which might not.

In any case, information gathering and synthesis, like any other part of the process, should reflect the needs, interests, and abilities of all stakeholders.

How do you gather and synthesize information?

There are a number of steps to gathering and putting together the information you need. Most of these can be group activities, part of the participatory process. The actual information gathering can be parceled out to specific individuals or sub-groups.

Decide what you need to know

Not surprisingly, the first step in gathering information is determining what information to gather. There are a number of areas to explore:

• Details about the issue. These might include its immediate and root causes; its general effects on individuals and communities; its consequences; its development through different stages; its history; and the history of attempts to address it.
• How the issue has been dealt with elsewhere. Best practices or approaches for which there is an evidence base; other approaches that have been at least partially effective; and what hasn’t worked, which may give you at least as much important information as what has.
• People who can help. This category encompasses experts in the field and people or organizations that have run or been involved in successful attempts to address the issue.
• Who is affected locally, and how. This really comprises two questions: a) What population groups – geographical, ethnic, cultural, racial, class, etc. – are particularly affected by this issue?  and b) What other groups are affected, but less visibly? These might include those who work with the first group(s) in the community (teachers, for example, or social workers), those who depend on them, and those on whom they depend.
• The importance of the issue to the community. Again, this implies a double question:
  • How important does the community perceive the issue to be? and
  • How much and in what ways does the issue actually affect the community as a whole?
• Community needs related to the issue. What has to be added to or removed from the community in order to improve the situation? What kinds of approaches will the community respond to or reject?
• Other context information. Community history, relationships among groups and individuals that might be relevant to your work, community culture, etc.
• Who, if anyone, has some influence or control over changing the situation. Public officials and other policymakers are often in this position. Business leaders, landlords, government enforcement agencies, schools, employers, hospitals and health personnel, and members of the affected group itself might also be in the position to change the situation (by learning new skills or changing practices).

Determine your likely information sources

As mentioned above, these encompass existing (i.e., published) sources and natural (i.e., experiential) examples. Published sources can be divided into scholarly, mass-market, and statistical, each of which can provide different information and a different perspective on the issue and attempts to address it.  Depending on what you decide you’re looking for, you might use all, or any combination, of these sources.

The single largest storehouse of information available is the Internet. Many scholarly articles are published online and accessible – often free, sometimes for a fee – to anyone who’s interested. Virtually all U.S. laws and regulations at every level of government are easily found, most on several websites. General knowledge on just about anything is widely available, as are lists of best practices and successful organizations and the websites of those organizations. Census data and other similar statistical information are also on view.  Add to these the information provided by such all-encompassing sources as Wikipedia (recently, for all its quirks, found to be just about as accurate across its million-plus entries as the Encyclopedia Britannica), and you have a nearly-bottomless well of fact and opinion to draw from.

As always, you have to be cautious: most of cyberspace is unedited, and the quality of information varies.  If you stick to reasonably reliable sites, you’re likely to find almost whatever you need, or at least directions to it.

Existing sources

Scholarly sources might include:

• Academic and some professional journals
• Books written for the academic market
• Doctoral dissertations - these are accessible to researchers through university libraries and some Internet sources
• Papers and reports delivered at academic and professional conferences - these are often available online, either on the authors’ websites or in e-published conference proceedings
• Occasional articles in respected mass-market scientific magazines, such as Nature or Scientific American
• Newspaper archives
• Direct contact with academics and other researchers who’ve done work on the issue you’re interested in, or who have conducted studies of attempts to deal with it
• Internet listservs and news groups relating to the issue or the field in question

Mass-market sources of information:

• Widely available books, often marketed as “self-help” or “life-changing,” to the public at large
• Articles in popular magazines, both those devoted to science or behavior and those of general interest
• Newspaper stories, often in Sunday magazine sections

Where to find statistical and demographic information:

• Census data - available on the web and at many libraries
• Community reports, such as community report cards, self-studies, and needs assessments, all of which should be obtainable through the appropriate municipal offices, and sometimes on the web as well
• Organizational and agency data, usually a matter of public record if the agency is public or publicly funded

In addition to these sources, the broadcast media often present stories about critical issues or about successful efforts to address them. In most cases, such stories only skim the surface, since they have to fit into short time slots (public broadcasting, on both radio and TV, breaks this mold more than other media outlets).  They can, however, serve as introductions to further research, raising the importance of one or more aspects of an issue, or providing information about effective programs that you can then contact.

Natural examples

Some of the more likely sources of natural examples:

• Program directors
• Friends or colleagues in the field
• Funders (particularly public agencies, because their transactions, including whom they fund and why, are a matter of public record)
• Leaders and members of community coalitions or partnerships
• Officials who coordinate community-wide efforts
• Members of the population most directly affected by the issue at hand
• Current or former participants in or beneficiaries of effective programs
• People who work in collaboration with programs – police, medical staff, teachers, etc.
• Key informants in the community
• Experts – some of them the same academics and other researchers referenced under scholarly sources – who have experience with your issue and efforts to address it
• Your own experience in the community

Don’t be afraid to range far and wide in your search for successful models or new ideas.  Step outside your own field and your own region, and see what’s been done elsewhere.  A model from social work or urban design might work in public health, or vice-versa.  There’s enough overlap among fields that deal with human health and development that you can often find exactly what you need in seemingly odd places.

Devise a plan for collecting information

There are a number of considerations here:

• Who will gather what information? As we’ve discussed, the ideal group is multi-sectoral and diverse in backgrounds and skills.  Information gathering should be assigned according to participants’ skills, interests, and contacts in the community. We’ve suggested, for instance, that scholarly sources might be mined by academics or other experienced researchers, while members of the affected population might be more successful in approaching key informants in the community. This doesn’t mean, however, that in a given group, these and other apparently logical roles couldn’t or shouldn’t be varied, depending on the individuals involved.
• How will the information be gathered? Another issue is just how the information will be gathered. Finding and reading written material is relatively straightforward: it’s in the library or on the web, and you can read and take notes on the relevant parts of it. Getting information directly from other people, however, can be more complicated. Will you engage in formal or informal interviews? In observation? Will you conduct surveys or public meetings? How will you contact people you don’t know – by letter, by phone, through mutual acquaintances? Your information-gathering methods will be determined by how much time you have, exactly what information you need, the depth of the information you need, and the abilities of the participants.
• What adjustments will be made for particular gaps in experience or skills?  People who don’t read, write, and/or speak the language proficiently may have to devise imaginative ways of recording information. Experienced researchers may have to translate scholarly writing for just about everyone in the group who isn’t an academic. In many cases, most or all of the group may need orientation or training before information gathering can begin. You’ll need to work out what the needs are as a group, and devise ways to meet them.
• What’s the timeline for information gathering? While information gathering should continue throughout the life of a project, the initial phase should have a time limit, so that action isn’t delayed for too long. The time limit depends on your time constraints, the seriousness and intensity of the issue, the community’s perception of urgency, and whether there are external time restrictions (student interns who are only available until the end of the summer, for instance.) Having a clear deadline will focus the group’s activities, and boost its efficiency.

Collect information

When your plan is completed, it’s time to put it into practice. You’ll have to conduct any trainings that are necessary, and make sure that all the relevant tasks are assigned appropriately. You may also want to set up regular meetings throughout the information-gathering process, in order to give the group the chance to review progress, make suggestions, and report on what they’ve been finding.  In addition to providing support for those new to research, these meetings, by providing a preview of the results of the process, will save everyone having to digest an overwhelming amount of information all at once.

Synthesize: Take it all apart

The process of synthesis involves breaking the information down into its component parts, sifting through those parts to see which fit together best for your situation, and then integrating them into an approach that is likely to work in your community.

There are usually three major areas to be considered:

• What’s known about the issue itself. What personal and environmental factors contribute to the problem? What are its root causes? Do you have the resources to address them, or are they beyond your scope (e.g., global economic forces or climate change)? Does the issue have a number of different effects, and if so, what are they?  What are the likely consequences for the community as a whole if the issue is not resolved?  (An environmental health risk can not only kill or sicken individuals, but might also affect business productivity, insurance availability and rates, hospital costs, the housing market, or even – as in the case of the Love Canal neighborhood in Niagara Falls, NY – the existence of a neighborhood or community itself.)
• The community context of the issue. What are the specific local effects of the issue. Exactly who is affected?  Exactly how are they affected? What are the consequences for those individuals? For their families, friends, neighbors, and others they have dealings with? For the community as a whole?  What has been the community’s experience with this issue in the past?  How, if at all, has it been addressed?  What local conditions would change if the issue were addressed, and how would they change?  Are there underlying conditions that have to change before the issue can be addressed?  Whose attitudes and/or behaviors need to change to have an effect on the issue (for example, among policy makers, those affected, or specific officials)?
• Successful and unsuccessful attempts to address the issue. These may have been gleaned both from the literature on best practices, and directly or at second hand from those involved in them. Here, it’s important to separate out the elements of various approaches. What specific procedures – methods and intervention components – were used? What kinds of training – feedback, role play, modeling, etc. – were provided to participants?  Was information provided to participants about when, why, and how to act? Were there positive or negative consequences that helped to establish or maintain change (or its opposite)?  Were environmental barriers, policies, or regulations put in place or removed?  What was the overall philosophy behind the approach?  What aspects of the issue did it address? What kind(s) of community was it tried in?  What population groups (in terms of culture, age, social class, etc.) were involved?  Who was the approach to benefit?  What were the specific results in the short term?  In the long term?  What makes a particular program, policy, or practice successful or unsuccessful? What events, if any, were critical, to success (or failure)?  What conditions – organizational features, participant characteristics, broader environmental factors – were critical?  Is there a model successful program? Is there a model unsuccessful program?

The existence of a model unsuccessful program doesn’t indicate that if you do the opposite of everything that program did, you’ll be successful.  Even if it failed spectacularly, much of the program may have been potentially effective, but one or two elements – the way participants were approached, recruited, or treated, a particular method – negated what could have worked.  By the same token, most elements of the program may have been fine, but its basic premise might have been mistaken or ineffective – “Just say no” as a way of preventing AIDS among teens, for instance.  It’s important to try to figure out why the program was unsuccessful.  A true model unsuccessful program is one that did everything wrong, but those are few and far between.

Lisbeth Schorr (Common Purpose) makes a useful distinction between “what works” and conditions under which what works actually works.  Sometimes the presence of a charismatic leader or champion motivates staff and/or participants to succeed. When the leader or original staff members leave, some such programs collapse, while others are able to renew themselves by careful hiring and a faithful implementation of what made it work.

In looking for programs to draw from, you need to understand the intervention components and elements that make those programs work.  Also, try to understand the conditions that allow an intervention to be successful.

Synthesize: Put it back together

Analyze the elements you’ve found to determine which of them would be appropriate for the situation and group you’re working with.

• What has been used specifically with your population in your circumstances?  Have the successful programs you’ve looked at been context-specific (i.e., intended for their specific communities and populations)? Can they be adapted to your context if they weren’t intended for it?
• What can be adapted, if it wasn’t originally aimed at your population?  (Techniques used with children or adolescents that could be modified for use with adults, for instance, or vice-versa.)
• What’s missing?  What aspects of the issue in your community are not addressed by what you’ve found?  Are they important enough that they need to be addressed?
• Does what you’ve found confirm or contradict what you thought you already knew?
• Are there factors in your particular situation that make the issue substantially different for you and your participants than for any other programs or approaches you’ve found out about?  How will you deal with that?
• What does your information tell you about the possibility of successfully addressing the issue’s root causes (e.g., income inequality, social exclusion, lack of power)?
• In general, did most or all successful programs direct their change efforts at the same group of people (policy makers, for example), or was there a variety?  If the latter, what do you think is most likely to work in your community?
• Perhaps most important, what’s your definition of success, and which of the programs you learned about came closest to achieving?  What components and elements of those programs addressed what’s needed in your community?

Answering these questions will give you a good sense of which components of other programs may work for you, and should also fit with what you already know to either give you ideas for new elements that you can add, or confirm (or warn you away from) ideas for new elements that you had already.

We don’t want to imply that simply taking a lot of different program components and playing mix-and-match will provide you with an effective way to address a community issue. You have to start with a clear framework informed by your vision and mission, and put together a program that’s coherent and makes sense.  All the elements have to fit; if they fit well enough, you’ll end up with a whole that’s greater than the sum of its parts. If the elements don’t fit together, or aren’t part of a program with a well-defined framework, the chances are you’ll end up with a mess.

Keep at it

Information gathering and knowledge synthesis should continue throughout the course of the program. While you may wait until the results of an initial evaluation to change something, you should always be looking for improvements and better approaches. No program or effort is perfect: everything can be improved. As long as you keep trying to learn more and grow in your understanding of your work, it will continue to get better. If you become complacent (i.e., you feel you know what you’re doing and can relax), your program may start to lose its effectiveness.

In Summary

Gathering the information that already exists about your issue and attempts to address it is one of the most important aspects of planning a program or evaluation. By putting together what’s known about the issue and the history of the successes and failures of various approaches to it, you can build a program structure that includes your own innovations and elements that have worked for others in similar situations. This synthesis also allows you to avoid ineffective approaches and to incorporate ideas and methods that have been particularly appropriate, culturally or otherwise, to the population and community you’re working with.

Information gathering and synthesis should continue throughout the life of the program. The more information you have, and the more carefully you put it together, the better your chances of implementing a successful program.