Section 7. Using Existing Data

Existing data can help you evaluate whether an intervention is working without requiring you to collect every piece of information yourself. When relevant, reliable data are already available, using them can save time and resources while strengthening your evaluation. It is important to have data to evaluate the effectiveness of an intervention.

To support an evaluation, existing data must be relevant to your population, issue, intervention, and evaluation questions. They also need to be available in a form you can analyze. You data may need to clean, combine, or otherwise prepare the data before use. This section explains how to find, assess, and use existing data thoughtfully.

WHAT IS EXISTING DATA?

Existing data is information that an organization, agency, researcher, or other source has already collected for reporting, program administration, research, or another purpose. Examples include administrative and program records, public datasets and dashboards, surveys, research studies, and community assessments. Some sources are updated regularly, while others provide a snapshot from a particular point in time.

 

COMMON SOURCES OF EXISTING DATA 

Some of the most common sources of data that may be useful for your evaluation include: 

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry 

Existing data is often available through online dashboards, open-data portals, reports, electronic record systems, downloadable files, or direct requests to the organization from which the data originated. Some information will still require a formal data request or data-sharing agreement, particularly when it contains confidential or sensitive information. 

TYPES OF EXISTING DATA YOU MIGHT LOOK FOR 

Depending on your evaluation questions, you might look for data about a specific population or geographic area, including: 

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population 

WHY COLLECT AND USE EXISTING DATA?  

There are sometimes good reasons to collect new data, including when the information you need is not available elsewhere. When you collect new data, you also have more control over who is included, what is measured, and how the information is gathered. However, primary data collection can be time and resource intensive. 

If the information you need, or something very close to it, already exists, there are several good reasons to find and use it. 

• Existing data can save substantial time and effort, especially when you need information about a large population, geographic area, or time period. They may also provide broader coverage or greater detail than your organization could collect on its own.
• Existing datasets may already have been cleaned, organized, analyzed, or documented by people with specialized expertise. This work can reduce the technical burden on your team, although you should still review the methods, definitions, and limitations before using the results.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The existing data you find may be more sweeping or more specific than what you’d be able to gather. They may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Existing data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for.  In cases like these, the use of data that already exist might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• The data may have been collected using established methods and systems. This can save time and strengthen your evaluation, but it does not guarantee that the data are free from bias, missing information, inconsistent definitions, or other limitations. You still need to understand how the data were collected.
• Existing data can help you examine change over time. Historical information can show whether an observed change is part of a longer trend and whether similar patterns appear in comparable communities, the state, or the nation. This perspective may be difficult to obtain through short-term data collection alone.
• Existing data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exist elsewhere. 

WHEN SHOULD YOU COLLECT AND USE EXISTING DATA? 

• When it’s available and accessible. This is the key question. If you know the data exist and can obtain appropriate access, they may be useful. If the data do not exist, are too difficult to locate, or cannot be shared because of privacy or other restrictions, you may need another approach.
• When it’s relevant. As with availability, the relevance of the data to what you’re trying to find out is a key issue. All the existing data in the world won’t do you any good if they don’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, existing data may be the only source of the information you need.
• When the data match your evaluation needs. Data should align with the population, place, time period, measures, and program elements you are studying. Select only sources that can help answer your evaluation questions; otherwise, the information may provide an incomplete or misleading picture. 

HOW DO YOU FIND AND USE EXISTING DATA? 

As you search for and use existing data, there are several questions you should ask. 

Are the data a good fit for your evaluation? 

Before using an existing source, look beyond whether it is easy to obtain. Ask whether it can answer your evaluation question and whether it can be meaningfully compared with any data you collect yourself. 

• Who and what do the data represent? Make sure the population, geographic area, and issue are relevant to your program.
• When were the data collected or updated? Older data can be useful for trends and context, but may not reflect current conditions.
• How were the data collected and defined? Review the data source’s methods, measures, and definitions. Two sources may use the same term—for example, “service received” or “graduation”—but mean different things.
• Are the data complete and sufficiently detailed? Consider missing data, small numbers, the level of detail available, and whether results can be broken down in ways that are useful and appropriate.
• What limitations or potential biases should you keep in mind? Administrative data reflect the people who use a service and the way staff record information; survey and research data may be affected by sampling, response rates, or measurement choices.
• Can you use the data ethically and securely? Follow data-sharing agreements and privacy requirements. Whenever possible, use aggregate or de-identified data rather than information that could identify individuals. 

WHAT INFORMATION DO YOU NEED, AND WHY? 

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants. 

Some possibilities include: 

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• Community-level indicators are specific, measurable signs of conditions or change across a community. Examples include program participation, services delivered, crime rates, or new cases of HIV. These indicators can help you track community trends and assess whether an initiative may be contributing to change. 

Community-level indicators describe trends for the community as a whole, rather than changes among individual participants. For example, an initiative to reduce alcohol use among young people might track weekend and nighttime single-vehicle crashes involving teenagers. Choose indicators because they are meaningful to the initiative and community—not only because data happen to be available. 

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include: 

• Demographics:
  • Demographic characteristics, such as age, race, ethnicity, gender identity, household composition, education, income, employment, housing, disability status, primary language, and geographic location
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level  
  • Other characteristics –primary language spoken, etc. 
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Alcohol and other substance use
  • Sexual behavior  
  • Various other behaviors – work habits, consumer patterns, etc. 
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Developmental and educational outcomes, such as completion of primary or secondary education and equitable participation by people with disabilities
  • General health and well-being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins) 

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes. 
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

Combining several years of data can produce more stable estimates when annual counts are small. Rolling averages—for example, comparing 1997–2000 with 1998–2001—can help show change over time while reducing year-to-year variation. The tradeoff is that multi-year estimates may make recent changes or program effects harder to detect.

Expanding the geographic area can also increase the number of observations and improve statistical stability. For example, you might use regional or state data when county-level counts are too small. However, broader estimates may not reflect local conditions and can hide meaningful differences among communities. 

Who is likely to have collected that information?

Public agencies, research organizations, universities, schools, health and human service providers, funders, advocacy groups, community organizations, and businesses may all maintain useful data. Start with the source most likely to collect information about your topic, population, or geographic area.

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

WHERE SHOULD YOU LOOK FOR EXISTING DATA? 

The question here is not only where to find existing information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble. 

Your own records and data 

Unless it’s brand new, your own organization should have administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information. 

Online data portals and websites 

Many public documents and datasets are available online. Start with the website or open-data portal of the government agency, organization, or institution most likely to have collected the data. In the U.S., federal, state, tribal, county, and municipal agencies often provide searchable reports, dashboards, or downloadable files. Similar sources are available in many other countries.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

Assess online data sources carefully. Favor information from the organization that originally collected or maintains the data, and look for clear documentation of methods, definitions, update schedules, and limitations. If a source is unfamiliar or a finding seems uncertain, compare it with other credible sources before using it. 

Go directly to the source

When data is not publicly available, contact the organization that maintains them. Explain what information you need, how you plan to use it, the level of detail required, and how you will protect privacy. Be prepared to discuss a data-sharing agreement, reasonable costs, or a mutually beneficial exchange of information or expertise. 

When requesting data from another organization for comparison, frame the conversation around shared learning rather than competition. Explain why the comparison is useful, how you will account for differences between programs or populations, and how findings will be interpreted and shared. An existing relationship built on trust and mutual respect can make this collaboration easier. 

Libraries

Libraries and librarians can help you identify credible datasets, government reports, scholarly research, archives, and specialized databases. They may also help you develop search terms, locate hard-to-find sources, and assess whether a source is authoritative. 

What are you planning to do with the data once you have it?

Plan how you will use the data before requesting or downloading them. Determine which variables, time periods, population groups, and level of detail you need. If you intend to compare existing data with information from your program, make sure the groups, definitions, measures, and collection periods are sufficiently similar to support a meaningful comparison.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the existing data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending. 

USING EXISTING DATA 

In many evaluations, you will combine existing data with new information collected from participants. Existing program or administrative records may also provide current information about participants, depending on what is recorded, how quickly the data are updated, and whether you have appropriate access. You may be able to find earlier data on participants to use as a baseline or data on a similar group to use as a comparison or control. 

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find existing data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both. 

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use existing data: 

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. A clearer picture of the community allows for a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved because of your work. 

In an evaluation, the intervention is the program, policy, service, or other action expected to contribute to change. The outcomes are the behaviors, conditions, or other results that may change. For example, a violence-prevention program is the intervention, while changes in violent behavior and related injuries are outcomes. 

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding. 

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group. 

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use existing data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change. The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find existing data that tells you whether the effects of your program continued, kept growing, or trailed off. 

In Summary

Existing data can help you understand community conditions, establish a baseline, identify trends, compare outcomes, and examine change over time. Useful sources include your own records as well as data maintained by public agencies, community organizations, health and human service providers, universities, funders, and other institutions. Select data that fit your evaluation questions, examine their quality and limitations, and follow all privacy and data-sharing requirements. Used thoughtfully, existing data can make evaluation more feasible and informative—especially for organizations with limited time and resources.