What do we mean by a design for the evaluation?
___The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know
___An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors
Why should you choose a design for your evaluation?
___So your evaluation will be reliable
___So you can pinpoint areas you need to work on, as well as those that are successful
___So your results are credible
___So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants
___So you can identify unintended consequences (both positive and negative) and correct for them
___So you’ll have a coherent plan and organizing structure for your evaluation
When should you choose a design for your evaluation?
___In the ideal, when you’re planning the program and evaluation, before you start any implementation
___In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters
Who should be involved in choosing a design for your evaluation?
___It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails
How do you select an appropriate design for your evaluation?
___Take into consideration the necessary research considerations
- Understand threats to internal validity ( whether the intervention produced the change)
-
History
-
Maturation
-
The effects of testing or observation on participants
-
Changes in measurement
-
Regression toward the mean
-
The selection of participants
-
The loss of data or participants
-
The nature of change
-
A combination of the effects of two or more of these
-
-
Understand threats to external validity (generalizability of your findings)
-
Interaction of testing or data collection and the program or intervention
-
Interaction of selection procedures and the program or intervention
-
The effects of the research arrangements
-
The interference of multiple programs or interventions
-
___Understand common research designs
- Pre- and post- single group design
- Interrupted time series design with a single group (simple time series)
- Interrupted time series with multiple groups (multiple time series)
- Control or comparison group
___Choose a design
___Consider your evaluation questions
___Consider the nature of your program
___Consider what your participants and staff will consent to
___Consider your time constraints
___Consider your resources