Search form

Section 3. Data Collection: Designing an Observational System

What do we mean by an observational system?

___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program

___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with

Why design an observational system?

___It can help you get reliable information

___It can help you find out exactly what you need to know, eliminating or reducing wasted effort

___It can ensure that observations are made

___It can make it easier to analyze your data

___It can help you avoid haphazard evaluation

___It will make it easier to justify your findings

___It can help you gain credibility with funders and policy makers

___It can let you pass on your practices with confidence

___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust

When should you design an observational system?

___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning

___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle

Who should design an observational system?

___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation

How do you design an observational system?

___Review your evaluation questions

___Decide what you need to observe to answer your questions:

  • Participants’ behavior
  • Someone else’s behavior
  • Conditions
  • Observations of results of behavior
  • Participants’ knowledge or attitudes
  • Someone else’s knowledge or attitudes
  • Goal attainment
  • Interactions
  • Program process or implementation (e.g., number of participants)

___Decide how the observations will be conducted:

  • Direct observation
  • Participant observation
  • Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
  • Second-hand reports, including interviews, journals, surveys, etc.
  • Electronic or mechanical observation
  • Tests or other similar observation tools
  • Public records and the like for community-level indicators

___Decide when you need to observe:

  • Pre- and post- observation
  • At regular intervals during the evaluation period
  • At irregular intervals during the evaluation period
  • At specific times during the evaluation period
  • Continuously

___Define and describe the behaviors, conditions, and/or events that observers should be concerned with

___Train observers in:

  • What it’s important to record, and why
  • The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
  • Effects of observation
  • Observer bias
  • Observer drift

___Devise checks for reliability and accuracy:

  • Use an external standard
  • Check for inter-rater reliability
  • Use random third-party checks

___Adjust the system for the next evaluation