What do we mean by an observational system?
___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program
___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with
Why design an observational system?
___It can help you get reliable information
___It can help you find out exactly what you need to know, eliminating or reducing wasted effort
___It can ensure that observations are made
___It can make it easier to analyze your data
___It can help you avoid haphazard evaluation
___It will make it easier to justify your findings
___It can help you gain credibility with funders and policy makers
___It can let you pass on your practices with confidence
___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust
When should you design an observational system?
___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning
___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle
Who should design an observational system?
___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation
How do you design an observational system?
___Review your evaluation questions
___Decide what you need to observe to answer your questions:
- Participants’ behavior
- Someone else’s behavior
- Conditions
- Observations of results of behavior
- Participants’ knowledge or attitudes
- Someone else’s knowledge or attitudes
- Goal attainment
- Interactions
- Program process or implementation (e.g., number of participants)
___Decide how the observations will be conducted:
- Direct observation
- Participant observation
- Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
- Second-hand reports, including interviews, journals, surveys, etc.
- Electronic or mechanical observation
- Tests or other similar observation tools
- Public records and the like for community-level indicators
___Decide when you need to observe:
- Pre- and post- observation
- At regular intervals during the evaluation period
- At irregular intervals during the evaluation period
- At specific times during the evaluation period
- Continuously
___Define and describe the behaviors, conditions, and/or events that observers should be concerned with
___Train observers in:
- What it’s important to record, and why
- The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
- Effects of observation
- Observer bias
- Observer drift
___Devise checks for reliability and accuracy:
- Use an external standard
- Check for inter-rater reliability
- Use random third-party checks
___Adjust the system for the next evaluation