Section 7. Collecting and Using Archival Data

Learn how to use existing information, both archival and secondary data, which can be much easier than collecting the data yourself.

 

You’re evaluating your teen pregnancy prevention program, and you’d like to know whether it will result in a reduction in pregnancy rates among young girls in the community. There are statistics on pregnancy rates for the state and county collected by the Public Health Service, but none at the community level, and none that separate the rates for girls under 16, the population you’re most concerned with.  You could do a community survey to try to find out the local rate, but there are two problems with that idea. The first is that you have neither the time nor the resources to conduct the survey, and the second is that you’re unlikely to get an accurate picture– there may be embarrassment in asking the questions or other reasons.

There might be another way of getting the information, however. A number of other agencies in the community work with youth, and one of them, or a combination, might have the figures you’re looking for. Rather than generating or collecting it yourself, you can save a great deal of time and trouble by using data that already exist.

We have previously discussed ways to find existing information to help you conduct a community assessment of assets and needs, but now our goal is somewhat different, because we’re seeking data that you can use to evaluate your work.This means that it needs to be in a form that can be analyzed, and may have to refer to a very specific population, issue, and/or method. As a result, it may be harder to find, and may have to be converted in some way once you do find it.  In many situations, however, using existing information still can be much easier than collecting the data yourself. In this section, we’ll try to help you make the use of archival data as easy as possible.

What are archival data?

Archival data refer to information that already exists in someone else’s files. Originally generated for reporting or research purposes, it’s often kept because of legal requirements, for reference, or as an internal record. In general, because it’s the result of completed activities, it’s not subject to change and is therefore sometimes known as fixed data.

Some researchers make a distinction between archival and secondary data. They see archival data as information specifically collected for bureaucratic procedures and the like – applications, reports, etc. – that can then be made usable for other purposes.  Secondary data refer to research information, collected as a result of studies and similar efforts, that can then be used by others either as comparison data or as part of new research.  For the purposes of this section, we’ll include both of these types of data in our discussion, and not distinguish between them.

Sources of archival data

Archival data can exist almost anywhere that information is collected.

Some of the most common sources (we’ll look at these and others later in more detail) are:

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

Archives are often stored as paper files or on electronic storage – computer disks, CDs, DVDs, etc. – and may include photographs and audio and video recordings as well. It may also take the form of encoded information expressed in numbers, or in computer language. Computer files, of course, may include various media and text, all in the same place.

Many organizations have archives so large that they store most of the material off-site, either with a data storage firm, or in their own or a rented facility. Some archives are made available on a website maintained by the government or other organization.

Types of archival data you might look for

As explained above, much of the data you’re likely to use for evaluation purposes will probably be more focused than data you’d use for an assessment of the level of a problem. Evaluation information would be more likely than assessment data to come in the form of study results, for example, than as narrative history or original documents. There’s a good deal of overlap: census data, for instance, could be used in both assessment and evaluation. In general, however, the possibilities below would refer to the types of data available, including information for a specific population or geographic area on:

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population

Why collect and use archival data?

There are sometimes good reasons for using original data, including that the information you need just isn’t available elsewhere. Additionally, if a researcher collects original data, he or she has more control what data are collected.

On the other hand, if the information you need, or something very close to it, already exists, there are several good reasons to find and use it.

• It’s easier and less time-consuming than collecting all the data yourself. This is probably the most obvious and most common reason for taking advantage of archival data. Especially if you’re looking for a large amount of information or information about a large group of people, you may be able to save yourself an enormous amount of time and trouble by using archival data.
• Archival data may have already been processed by people with more statistical expertise. Unless you’re a statistician or a health or human service researcher with an advanced degree (and often not even then), the chances are that you don’t have a flawless grasp of data analysis. You can hire someone or find a volunteer to help you, but if the hard work has already been done, it will make your work that much easier.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The archival data you find may be more sweeping or more specific than what you’d be able to gather. It may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Archival data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for  In cases like these, the use of pre-existing data might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• It may eliminate the need to correct for problems, such as improper sampling, lack of inter-rater reliability, or observer bias.
• Archival data allows the possibility of looking at the effects of your work over time. Is the change in your population part of a trend that seems to be reflected in data from a similar population or the entire state or nation? You may not have the capacity to collect data over a long enough period to answer such questions, but if the data already exist, it makes longer-term analysis possible.
• Archival data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exists elsewhere.

When should you collect and use archival data?

• When it’s available. This is the key question. If you know the data exist and you can get access, use it. If the data doesn’t exist, if finding it would take more time and effort than it’s worth, or if you have no access to it, then it’s not possible.
• When it’s relevant. As with its availability, the relevance of the data to what you’re trying to find out is a key issue. All the archival data in the world won’t do you any good if it doesn’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, archival data may be the only source of the information you need.
• When it can inform your evaluation. There are large amounts of archival data available almost everywhere. The mere fact that it exists doesn’t mean that it will do you any good. You have to be selective about what you gather and use. Make sure it is actually what you need, that it refers to the population and/or other elements of your program that will make it truly useful to you.  If not, you’ve not only wasted your effort, but the resulting evaluation won’t give you a realistic picture of your work and how to improve it.

How do you collect and use archival data?

As you search out and collect archival data, there are several questions you should ask.

What information are you looking for and why?

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants.

Some possibilities include:

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• General information on the population and/or the community you’re working with. You may want to see how well various characteristics of your participants match those of the general population or you may simply want to understand the context of the evaluation better. You may also be looking for information to choose community-level indicators. Community-level indicators are specific, measurable units that help us determine success of an initiative or intervention in the community; examples of community-level indicators include program participation rates, services delivered, levels of crime, or new cases of HIV/AIDS.

Community-level indicators show trends for the community as a whole.  You can often use them to find out whether your efforts have had any effect in the community.  If, for example, you are conducting a program to reduce alcohol use among youth, one indicator of success might be a reduction in the number of weekend and nighttime one-car crashes involving teens. In many cases, you might choose community-level indicators according to the data that’s available.  For example, if there are reliable figures for exactly the kinds of car crashes described, then using those figures as a community-level indicator would probably make sense.

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include:

• Demographics:
  • Demographic statistics – age, race, gender, ethnicity
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level Other characteristics –primary language spoken, etc.
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Substance use and abuse
  • Sexual behavior Various other behaviors – work habits, consumer patterns, etc.
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Development outcomes (e.g. those completing primary education, high school; those with disabilities)
  • General health and well being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins)

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes.
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

One way to increase the reliability of data where you are dealing with small data sets is to combine multi-year data (for instance, results of cancer deaths in a community for three years instead of one).  A drawback to this option is that looking at multi-year data limits the ability to monitor program interventions and identify new trends.  Rolling year averages (e.g., looking at data for 1997-2000 one year, and 1998-2001 the following year) may overcome this drawback and should is an option that should be considered.

Another way to decrease the possibility of statistical instability is to expand the geographic area you are investigating by looking at regional health assessments conducted by collaborating neighboring jurisdictions, or in the example above, expanding from county to state.  A drawback to this option is that you may then be examining results for a geographical area that does not necessarily apply to your assessment.   Analyzing data at the regional level may also mask interesting local variations in the data.

Who is likely to have collected that information?

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

Where should you look for archival data?

The question here is not only where to find archival information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble.

Your own archives

Unless it’s brand new, your own organization should have an archive of administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information.

The Internet

Most public documents are either on the Web or can be found and/or ordered through a website. The place to start is usually the website of the government agency most likely to have collected the data. The Resources portion of this section contains a list of U.S. government websites. In the U.S., states and most cities and towns have websites as well, with links to state or municipal agencies and departments. (The URL’s for all state websites take the same form: http:// www .[state abbreviation].gov.  Municipal websites can easily be found by searching the name and state of the community.) States or provinces and communities in most of the developed, and much of the developing, world have websites as well.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

As always when using the Internet, you should be cautious about where you find your information. There are enormous numbers of reliable websites...and huge numbers of unreliable ones as well.  If you’re not sure of a website or of the information you get from it, try to find that information elsewhere as well. In general, you can rely on websites when you know where they get their information, and when you trust the reputation and integrity of the site’s owner.

Go directly to the source

Often, the best way to find information from health and human service organizations, schools, and businesses, as well as from advocates and community activists, is to go to them directly.  If you do, be prepared to explain exactly what you’re looking for, what you plan to use it for, and what you can offer in return. Unless the organization is willing to let you comb through its files – confidentiality is often a barrier to that – someone will have to spend some time finding what you need.  It’s only fair to offer something in return, whether it’s payment, data analysis services, advocacy for the cause, or something else the other organization needs.

If you’re asking another similar organization for data so you can use it as a comparison or control group, the request has to be extremely tactful.  In a sense, you’ll be telling the staff of that organization that you expect your results to be better than theirs.  Depending upon how they see their work – and how they perceive you and your organization – they may take this as an opportunity to find better methods to serve their participants, or as a grave insult.  If it’s the latter, they’re hardly likely to agree to the use of their data.  You’ll have to frame the request in the right way, and offer a good exchange as well.  It will help if you’re dealing with an organization with which you already have a good relationship of mutual respect.

Libraries

Librarians have always been world-class experts at finding what library users needed.  With current technology, they’ve become even better. Many have an encyclopedic knowledge of not only what’s available in the library itself, but what’s on the web as well.  They may be familiar with sources of archival information you’d never think of, and be able to help you find what you need quickly and with minimum effort.  When in doubt, head to (or communicate with) an available library.

What are you planning to do with the data once you have it?

This question has to do not only with what form you need the data in, but also just what data you actually need. If you’re planning to use it as a comparison to the group participating in the program you’re evaluating – whether as a formal control group or as baseline data – you’ll need information on the variables you’re planning to look at, as they relate to the population you’re working with, or at least a population that’s reasonably similar. If, for example, you’re evaluating a chronic disease prevention program intended to benefit Latinos, and you’ve found archival data on physical activity and nutrition among Native Americans, you can’t compare your results with those of the archival data because the groups are likely to be too different.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the archival data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending.

Using archival data

It’s difficult to imagine evaluating a program or approach without actually collecting your own data on participants.  You might be able to find data on those participants from an earlier time, which you can then use as a baseline. You might be able to find appropriate data on a similar group that you can use as a comparison or control. But you can’t find data elsewhere on what those participants are currently experiencing, and that’s what you’re evaluating in almost every case.

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find archival data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both.

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use archival data:

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. Along with a clearer picture of the community goes a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved as a result of your work.

There are two kinds of variables (things that may change) in research.  An independent variable or intervention is a program, treatment, method, system, or other action or condition set up by the researcher to see if it will create change and improvement.  A dependent variable is a behavior, condition, or other element that may change as a result of the independent variable.  A violence prevention program, for example, is an independent variable that may change community members’ engagement in violent behavior and associated injuries, the dependent variables.

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding.

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group.

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use archival data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change.The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find archival data that tells you whether the effects of your program continued, kept growing, or trailed off.

In Summary

Most government agencies and departments, community-based health and human service providers, advocacy organizations, universities, and many other entities keep archival records of information. You may be able to use these as part of the data for your evaluation, saving time and trouble. Especially for small organizations with limited resources, the use of archival data can make it possible to produce an evaluation that provides the information needed to accurately assess a program’s effectiveness and make the changes necessary to improve it.