Section 5. Collecting and Analyzing Data

In previous sections of this chapter, we discussed studying the issue, choosing a research design, and creating an observational system for gathering information for your evaluation. The next step is to collect and analyze your data—examining what the information shows and what it may mean for your work. This section focuses on how to carry out that process.

What do we mean by collecting data?

Collecting data means putting your information-gathering plan into practice. You have already decided how you will obtain information—through direct observation, interviews, surveys, assessments, administrative records, or other methods—and now you and others involved in the evaluation need to carry out that plan. Effective data collection also requires clear definitions, appropriate timing, consistent procedures, and careful documentation so that the information you collect is useful for answering your evaluation questions.

Recording and organizing information may take different forms depending on the type of data you are collecting. Your data-collection procedures should also reflect how you plan to analyze and use the information. Whenever possible, information should be recorded during data collection or soon afterward so that important details are not lost. Data should also be stored securely and in ways that protect participants' privacy and confidentiality.

Some of the steps involved in preparing collected information may include:

• Bringing together information from all relevant sources and observations
• Creating secure backups of forms, records, recordings, and other materials to reduce the risk of loss, accidental deletion, or damage
• Entering narratives, numbers, and other information into appropriate software, databases, spreadsheets, or data-management systems
• Preparing quantitative information for analysis. This may include entering observations into tables or spreadsheets and calculating descriptive statistics such as the mean, median, mode, percentages, rates, or other relevant measures
• Transcribing relevant portions of audio or video recordings when text versions are needed for analysis
• Coding qualitative information by identifying categories, concepts, themes, or other features that make the information easier to organize and analyze
• Organizing data in ways that support your evaluation questions. You might group information by outcome, participant or group, time period, activity, location, implementation stage, or another relevant category. Looking at data in more than one way can also help identify relationships and differences that might otherwise be missed.

Research often distinguishes between independent variables and dependent variables. An independent variable is the program, intervention, policy, method, or condition whose effects you want to examine. A dependent variable is an outcome or condition that may change in response to that intervention. For example, a smoking-cessation program may be treated as the independent variable, while changes in participants' smoking behavior may be one of the dependent variables.

What do we mean by analyzing data?

Analyzing information means examining it systematically to identify patterns, relationships, trends, differences, and other findings that can help answer your evaluation questions. Analysis may involve statistical procedures, comparisons across groups or time periods, visual inspection of trends, or careful interpretation of interviews, observations, documents, and other qualitative information. The purpose is to develop the most accurate and useful understanding possible of your program, its implementation, its outcomes, and the conditions that may have influenced those outcomes.

Evaluations commonly use two broad types of data, although not every evaluation will include both. Quantitative data are expressed as numbers and can be summarized or analyzed mathematically. Qualitative data include descriptions, experiences, observations, quotations, interpretations, and other information that helps explain meaning and context. Each provides different kinds of insight, and they often work especially well when used together.

Quantitative data

Quantitative data are often collected directly as numbers. Examples include:

• The frequency, rate, duration, or intensity of particular behaviors, events, or conditions
• Assessment or test scores related to knowledge, skills, health, or other outcomes
• Survey responses, such as ratings of satisfaction, stress, confidence, access, well-being, or reported behavior
• Counts, rates, or percentages describing a population—for example, the percentage of people diagnosed with a particular health condition, the unemployment rate, languages used in a community, age distribution, or levels of educational attainment

Information collected in other forms can sometimes be converted into quantitative data. Researchers might count how often particular themes or events appear in interviews or records, for example, or use a rating scale to describe the intensity or frequency of an observed condition. Community initiatives may also document the number and type of environmental, organizational, programmatic, or policy changes connected to their work. Whether converting information into numbers is useful depends on what you are trying to understand and whether doing so preserves the meaning of the original information.

Quantitative information may be summarized with descriptive statistics such as frequencies, percentages, rates, averages, medians, and ranges. More advanced statistical procedures can examine differences between groups, changes over time, associations among variables, and the strength of evidence for particular conclusions. Numerical analysis can be powerful, but numbers do not interpret themselves. The quality of the findings still depends on what was measured, how measurements were collected, whether the sample represents the people or conditions of interest, and whether the analysis is appropriate for the evaluation design.

Qualitative data

Qualitative information provides detail about people's experiences, meanings, perspectives, interactions, and the context in which events occur. It can help explain findings that numerical measures alone may not capture. For example, a test score may indicate how a student performed, while an interview or observation may help explain the student's experience, confidence, barriers, learning environment, or response to the result.

Qualitative information can sometimes be summarized numerically—for example, by counting how often themes appear or asking participants to rate dimensions such as satisfaction, importance, accessibility, or usefulness. However, converting qualitative information into numbers can reduce some of its context and meaning.

Rating scales illustrate this limitation. Two people may both select the same numerical rating while having very different reasons for doing so. A participant who rates a program a “3” might appreciate its content but find the schedule difficult, while another might like the schedule but feel the content does not meet their needs. The number alone cannot explain those differences.

Qualitative information can help explain why particular methods work well or create barriers, how participants experience a program, whether an approach fits the local context, and what participants consider most important. It can also reveal patterns or factors that were not anticipated when the evaluation was designed.

Whenever appropriate, collecting both quantitative and qualitative information can provide a more complete understanding of a program and its outcomes.

Neither quantitative nor qualitative analysis is completely free from human judgment. Quantitative analysis uses standardized mathematical procedures, but decisions about what to measure, which people or data to include, which statistical procedures to use, and how findings are interpreted still affect the results. Qualitative analysis involves interpretation more directly, making it especially important to use clear procedures, consider multiple perspectives, document how conclusions were reached, and reflect on how researchers' assumptions and experiences may influence interpretation.

Why should you collect and analyze data for your evaluation?

Not every organization has the resources to conduct a large or highly technical evaluation. Smaller community-based organizations may use simpler evaluation approaches that still provide valuable information for understanding and improving their work. Regardless of the level of complexity, systematic data collection and interpretation can strengthen decision-making and future planning.

• Data can show whether meaningful changes occurred in the outcomes you hoped to influence. Collecting information before, during, and after an intervention can help you determine whether conditions changed and whether the pattern of change is consistent with the goals of the program.

In statistical analysis, the term statistical significance has a specific meaning. Researchers often use a threshold such as p < .05 to indicate that the observed result would be relatively unlikely under a particular statistical assumption if there were actually no effect or difference. Statistical significance should not be confused with practical importance, the size of an effect, or the percentage of participants who benefited.

Statistical results should therefore be interpreted together with the evaluation design, effect size, confidence intervals when available, sample size, data quality, practical significance, and other evidence. A statistically significant result does not by itself prove that a program caused a particular outcome.

• Analysis can identify factors associated with differences in outcomes. For example, participants who have access to a peer-support group may show different outcomes from participants who do not. Findings like these can help identify implementation features that deserve closer examination.
• Analysis can identify relationships among different factors. Statistical procedures may reveal correlations or associations among variables. Two outcomes might increase together, decrease together, or move in opposite directions. These relationships can generate useful questions, but correlation alone does not establish that one variable causes another.
• Analysis can help explain why your work was more or less effective than expected. Combining quantitative and qualitative information can help identify implementation strengths, accessibility barriers, contextual conditions, participant experiences, or other factors that influence outcomes.
• Analysis can provide credible evidence for stakeholders. Participants, community members, governing boards, funders, and other partners may want to understand what was accomplished, what challenges remain, and what the organization has learned. Evidence about implementation, intermediate outcomes, and longer-term outcomes can support accountability and future planning.
• Regular use of data demonstrates a commitment to learning and improvement. Reviewing information about progress, barriers, and outcomes can help organizations act as responsible stewards of community resources.
• Sharing what you learn can strengthen work beyond your own program. Findings about effective strategies, implementation challenges, adaptations, and community context may help other organizations develop or improve related efforts.

When and by whom should data be collected and analyzed?

Whenever possible, data collection should begin before or when the program starts so that you can establish baseline information. Collecting information before implementation can also reveal trends that were already occurring. Data collection may continue throughout the program and, when longer-term outcomes matter, during an appropriate follow-up period after the formal intervention ends.

Data can be analyzed at different points depending on the purpose of the evaluation. A primarily summative evaluation may emphasize analysis after the main data-collection period so that findings can be considered together. A formative evaluation may examine information throughout implementation so that emerging findings can inform adjustments. Many evaluations combine these approaches by reviewing data periodically while also conducting a more comprehensive analysis at the end of an evaluation cycle.

Who collects and analyzes information depends on the methods being used, available expertise, resources, ethical requirements, and the degree of participation built into the evaluation.

Possible approaches include:

• Partnering with or hiring an external evaluator, researcher, university program, or other person with relevant expertise
• Conducting a less technical but systematic evaluation using descriptive statistics, participation records, interviews, feedback, and other information that can still provide useful insight into the program
• Building staff or community evaluation capacity through training, courses, technical assistance, or mentoring
• Collecting information internally and partnering with someone who has appropriate expertise to conduct or review more specialized analysis
• Using qualitative methods when they are appropriate to the evaluation questions. Qualitative information may be especially valuable for understanding participant experiences, implementation, accessibility, barriers, and context, even when it cannot by itself answer every question about program effectiveness
• Using an appropriate comparison strategy when your design and ethical circumstances support one. Comparisons can strengthen an evaluation, but conclusions should still take into account baseline differences, sample size, uncertainty, and other factors rather than relying only on raw percentages.

In a participatory evaluation, community members, program participants, staff, and other stakeholders can contribute to both data collection and interpretation. Some forms of measurement may require specialized training, while other information may be best collected or interpreted by people with lived experience or deep knowledge of the community. Even when statistical analysis is conducted by a specialist, community participants can play an important role in interpreting what the findings mean, identifying missing context, and deciding how the results should inform future action.

How do you collect and analyze data?

Whether your evaluation uses highly formal research methods or a simpler community-based approach, several basic steps can help you collect and analyze information systematically.

Implement your measurement system

We've previously discussed designing an observational system to gather information. The next step is to put that system into practice.

• Clearly define and describe the measurements or observations you need. Definitions should be specific enough that different data collectors understand what is being measured and record information consistently.
• Select and prepare data collectors. Depending on the evaluation, they may need training in observation, interviewing, confidentiality, data security, accessibility, informed consent, or inter-rater reliability.
• Collect information at appropriate times and for an appropriate period. This may include reviewing existing records, conducting interviews, administering surveys, facilitating focus groups, observing program activities, or gathering other forms of data.
• Record information using agreed-upon procedures. Depending on the setting, this may involve secure digital forms, databases, spreadsheets, written records, audio or video recordings, journals, activity logs, or other appropriate formats.

Organize the data you've collected

How you organize information depends on your evaluation questions and how you plan to analyze it.

• Enter and store information in an appropriate system. This might include a database, spreadsheet, qualitative analysis program, secure document system, or GIS (Geographic Information Systems) platform. Use appropriate privacy and security protections, particularly when data include personally identifiable or sensitive information.
• Transcribe audio or video recordings when needed. Transcription can make recorded information easier to search, code, compare, and analyze. Review transcripts for accuracy and protect identifying information appropriately.
• Score assessments or tests according to the procedures appropriate for the instrument and record the results accurately.
• Sort information in ways that support your questions. You might organize data by participant group, outcome, activity, location, time period, implementation stage, demographic characteristic, or another relevant category.
• When useful and appropriate, summarize qualitative information quantitatively. This might include counting how often particular themes, experiences, or barriers are mentioned. Preserve the original context where it is important for understanding what those counts mean.

Conduct graphing, visual inspection, statistical analysis, qualitative analysis, or other appropriate operations on the data

Statistical procedures can help examine whether outcomes changed, whether groups differed, whether trends emerged over time, and whether particular variables were associated with one another. The type of analysis should match the evaluation design, the quality and amount of available data, and the questions you are trying to answer.

Useful forms of analysis may include:

• Counting, graphing, and visually examining frequencies, rates, or other measures over time
• Looking for meaningful changes in trends, such as clear increases, decreases, or shifts following implementation of a program or policy
• Calculating descriptive statistics, such as the mean, median, mode, percentages, rates, or ranges
• Analyzing interviews, conversations, journals, and participant observations to understand experiences, implementation, and changes over time
• Identifying themes and patterns in qualitative information. When many participants describe similar barriers, supports, experiences, or concerns, those patterns may help explain how the program is functioning and where changes may be needed.
• Comparing results with previously established goals, benchmarks, or implementation expectations

Take note of significant, meaningful, or unexpected results

Findings may be statistically significant, practically important, unexpected, or especially meaningful to participants and communities. These are not necessarily the same thing. A statistically detectable difference may be too small to matter in practice, while an important community-level change may not be captured well by a conventional statistical test.

Types of findings you may want to examine include:

• Changes within individuals, groups, organizations, or communities over time. Repeated measurements can help show whether outcomes changed after an intervention was introduced. Staggered implementation across people, groups, or sites may strengthen confidence that observed changes are associated with the intervention.
• Differences between or among groups. When groups are appropriately comparable, differences in outcomes may provide evidence about program effects. Interpretation should consider how groups were formed, whether they differed at baseline, and the uncertainty around the estimates.
• Statistically significant changes. Statistical tests may indicate that an observed difference is unlikely under a particular null hypothesis. Whether that difference was caused by the intervention depends on the evaluation design and other evidence.
• Correlations and associations. Variables may change together in ways that suggest potentially important relationships. These findings can generate new questions and hypotheses, but should not automatically be interpreted as evidence that one variable causes another.

Correlation does not establish causation. Two variables may change together because one influences the other, because the relationship works in the opposite direction, because both are affected by a third factor, or because the association occurred for another reason.

For example, two outcomes may both increase as young people get older, even though neither causes the other. On the other hand, a strong and repeated association between a medication and an adverse health outcome could provide an important signal that deserves additional investigation. The appropriate response is to examine the relationship carefully rather than assume a causal explanation from correlation alone.

• Patterns. Both quantitative and qualitative data may reveal patterns by geography, access to resources, language, age, race, ethnicity, disability, income, program location, or other relevant characteristics. Such patterns should be explored carefully rather than interpreted as inherent characteristics of a group. They may reflect differences in exposure, opportunity, access, policy, environment, discrimination, or other contextual factors.
• Findings with clear practical importance. Some results may stand out even before advanced analysis. For example, participants in a health program might show meaningful improvements in blood pressure, physical activity, or well-being compared with an appropriate comparison group. Conversely, consistently unchanged outcomes may signal that the program needs adjustment, is not reaching people who could benefit, or is measuring the wrong outcomes.

Important findings do not always answer only the question of whether the program “worked.” Evaluation may reveal that a community-wide issue is concentrated in particular neighborhoods, that some groups experience greater barriers to services, or that outcomes differ according to access to resources or environmental conditions. Examining findings from multiple perspectives can help organizations improve their programs while also identifying broader conditions that may require attention.

Interpret the results

Once you have organized and analyzed the information, the next step is interpreting what it means. A common evaluation question is whether the program contributed to meaningful change. In research terms, this involves examining the relationship between the intervention and the outcomes it was intended to influence while considering other possible explanations.

Several broad patterns of findings are possible:

• The program contributed to the outcomes you hoped to achieve. Evidence may show meaningful improvements associated with implementation, particularly when findings are consistent across multiple measures or stronger than those seen in an appropriate comparison group.
• The program does not appear to have produced the intended outcomes. The evaluation may show little or no meaningful change in the outcomes being measured.
• The program may be associated with harmful or unintended outcomes. These findings require careful interpretation. For example, an increase in reported incidents of interpersonal violence after a program begins could indicate an actual increase, but it could also reflect improved screening, increased awareness, greater trust in reporting systems, or better access to support.
• The program may produce several different effects.
  • Some effects may be beneficial. A youth violence-prevention initiative, for example, might reduce violent incidents while also improving school engagement or participants' sense of safety and belonging.
  • Some effects may be neutral or unrelated to the program's central goals.
  • Some effects may be harmful or unintended. A strategy that improves one outcome might create new barriers, increase disparities, or have unintended effects in another area.
  • Effects may also be mixed, differing across outcomes, participants, settings, or time periods. Understanding who benefits, who does not, and under what conditions is often as important as calculating the average effect.

If the evidence suggests that the program is accomplishing its goals, continue examining how it might be strengthened, sustained, or adapted. Effective programs should continue learning rather than assuming that current results cannot be improved.

Strong community programs are dynamic. They use evaluation findings to continue learning and adapting rather than treating initial success as the end of the improvement process.

If the evaluation shows limited, uneven, or harmful outcomes, interpretation becomes especially important. Is the underlying approach inappropriate, or is a promising approach being implemented inconsistently? Are there organizational barriers, accessibility issues, resource constraints, policy conditions, discrimination, or other structural factors influencing outcomes? Are some groups benefiting while others face barriers? Should particular components be modified, expanded, removed, or redesigned?

Careful interpretation can help answer questions like these. Associations among variables may generate hypotheses about factors contributing to outcomes, but those relationships should be investigated rather than treated automatically as causal. Qualitative information can be especially valuable for understanding why participants experience a program differently and for identifying supports, barriers, or contextual conditions that are not apparent in numerical data.

For example, if participants experience ongoing health challenges, qualitative information might identify barriers related to food access, housing conditions, transportation, affordability of care, language access, previous experiences with health systems, work schedules, caregiving responsibilities, cultural preferences, or the availability of trusted and responsive providers. Interpretation should begin with participants' own experiences rather than assumptions about their knowledge, culture, or choices.

Once your information has been organized and analyzed, both numerical findings and qualitative evidence should be considered together using careful reasoning and critical thinking. Numbers may show that change occurred, while participant experiences, program records, and contextual information may help explain what contributed to it and why.

Analysis and interpretation bring the evaluation process back to where it began: learning how to improve the work. Use what you have learned to adjust the program, continue monitoring outcomes, evaluate those changes, and repeat the process as needed. Ongoing evaluation can help strengthen programs while also identifying organizational, environmental, policy, and systems-level changes that support healthier and more equitable communities.

Understanding an initiative requires attention to context, relationships, culture, history, systems, and participants' lived experiences. In that sense, evaluation often involves learning not only whether something changed, but how people understand and experience that change.

In Summary

The heart of evaluation is collecting meaningful information about a program or intervention and analyzing that information to understand implementation, outcomes, strengths, limitations, and opportunities for improvement.

Quantitative data—information expressed numerically—can be summarized through graphs, descriptive statistics, and more formal statistical analysis. These methods can help identify changes, differences, trends, and associations among variables. Their interpretation should always take the evaluation design, data quality, uncertainty, and practical significance into account.

Qualitative data—including interviews, observations, participant experiences, descriptions of conditions, documents, and other contextual information—can help explain how and why outcomes occurred, what participants experienced, what barriers or supports were present, and what may need to change.

Once you have learned from the information you collected, use those findings to strengthen the next cycle of planning, implementation, and evaluation. Continued data collection, reflection, and analysis can help keep community programs responsive, effective, equitable, and accountable to the people they are intended to serve.