What do we mean by an observational system? ___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program ___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with Why design an observational system? ___It can help you get reliable information ___It can help you find out exactly what you need to know, eliminating or reducing wasted effort ___It can ensure that observations are made ___It can make it easier to analyze your data ___It can help you avoid haphazard evaluation ___It will make it easier to justify your findings ___It can help you gain credibility with funders and policy makers ___It can let you pass on your practices with confidence ___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust When should you design an observational system? ___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning ___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle Who should design an observational system? ___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation How do you design an observational system? ___Review your evaluation questions ___Decide what you need to observe to answer your questions: Participants’ behavior Someone else’s behavior Conditions Observations of results of behavior Participants’ knowledge or attitudes Someone else’s knowledge or attitudes Goal attainment Interactions Program process or implementation (e.g., number of participants) ___Decide how the observations will be conducted: Direct observation Participant observation Self-reports, including individual and group interviews, focus groups, journals, surveys, etc. Second-hand reports, including interviews, journals, surveys, etc. Electronic or mechanical observation Tests or other similar observation tools Public records and the like for community-level indicators ___Decide when you need to observe: Pre- and post- observation At regular intervals during the evaluation period At irregular intervals during the evaluation period At specific times during the evaluation period Continuously ___Define and describe the behaviors, conditions, and/or events that observers should be concerned with ___Train observers in: What it’s important to record, and why The definitions and descriptions of the behaviors, conditions, events, or situations to be observed Effects of observation Observer bias Observer drift ___Devise checks for reliability and accuracy: Use an external standard Check for inter-rater reliability Use random third-party checks ___Adjust the system for the next evaluation
