Table of Contents >
   Part J. Evaluating Community Programs and Initiatives
      Chapter 36. Introduction to Evaluation >
         Section 5. Developing an Evaluation Plan >
             Examples - Real world situational examples. >

Developing an Evaluation Plan

  

Examples

 Contributed by Chris Hampton Edited by Vincent T. Francisco and Bill Berkowitz

Example 1: Table of Program Components and Elements

Here are a few of the items from the table of program components and elements for a fictional teen pregnancy prevention program:

Risk/ Protective Factors

Program Components (Strategies)

Program Elements (Tactics)

Targets and Agents of Change

Mode/Context of Delivery

Risk factor: Unsupervised activities by teens

Provide alternative activities

  • After-school activities (sports programs, hobby /craft classes, movies, etc.)
  • Weekend "lock-ins", non-alcohol teen dance club
  • Daytime activities during school holidays

Targets:

  • youth

    • Agents:

  • youth
  • teachers
  • parents
  • Group sessions
  • Workshops
  • Meetings

    • Risk factor: Lack of access to contraceptives

    Increased access top health services

    • Extend hours at school health clinics
    • Increase condom distribution by county health department
    • Brown bag program at area pharmacies (teens wishing to buy contraceptives are able to call ahead and pick them up in a plain brown bag)

    Targets:

  • youth

    • Agents:

  • health providers
  • Walk-in clinics
  • Health department
  • Businesses

    • Protective factor: Involvement in the faith community

    Encourage youth involvement in the faith community

    • Increase church-sponsored youth activities
    • Institute church-based sexuality education
    • Church-sponsored parent-child communication sessions

    Targets:

  • youth
  • parents

    • Agents:

  • youth
  • parents
  • religious leaders
  • Discussions
  • Meetings
  • Educational sessions
  • Workshops

    • (Thanks to Adrienne Paine-Andrews of the Work Group on Health Promotion and Community Development for her assistance with the above table).


    Example 2: Evaluation Questions and Methods

    Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (Y.E.S.), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally , and interact with positive role models:

       

    Type of Data Collection*

    Experimental Design**

    Key Evaluation Question(s)

    Type of Management Information and Evaluation Measure(s)

    Survey/
    Scale

    Structured Interview

    Self Report/ Log

    Direct Observation

    Archival Records

    Case Study

    Pre-Posttest Control Group

    Time Series

    A.Planning and Implementation Issues

    A. Descriptive and Process Measures

    		               

    1.Who participates? Is there diversity among the participants?

    1.Basic demographic data (e.g., age, gender, disability, race, economic status , etc.) on participants.

    X

    X

    		     

    X

    		   

    2.Why do participants enter and leave the program?

    2.Information on participants? reasons for entering and leaving the program.

    X

    X

    		     

    X

    		   

    3.Are there a variety of service and alternative activities generated?

    3.Frequency of different types of service provided and alternative activities generated.

    		   

    X

    X

    X

    X

    		   

    B.Assessing Attainment of Objectives

    B. Outcome Measures

    		               

    1.How many people participate?

    1.Number of people participating.

    		   

    X

    X

    X

    X

    		 

    X

    2.How many hours are participants involved in service and alternative activities?

    2.Number of hours of participation by type of activity

    		   

    X

    X

    X

    X

    		 

    X

    3.How many people are trained in cultural awareness and sensitivity?

    3.Number of people trained in cultural awareness and sensitivity.

    		   

    X

    X

    X

    X

    		 

    X

    C.Impact on Participants

    		                 

    1.How do behavior and attitudes change as a result of participation in the program ?

    1.Changes in programs, policies, and practices of affiliated organizations, and other outcomes.

    X

    X

    X

    X

    X

    		 

    X

    X

    2.Does participation affect the incidence of problems in living?

    2.Incidence of problems in living (e.g., substance use, gang involvement)

    X

    X

    		   

    X

    		 

    X

    		 

    3.Are participants satisfied with the experience?

    3.Satisfaction ratings from participants.

    X

    		       

    X

    		   

    D.Impact on Community

    		                 

    1.What resulted from the program?

    1.Changes in programs, policies, and practices of affiliated organizations, and other outcomes.

    X

    X

    X

    X

    X

    		   

    X

    2.Do the benefits of the program outweigh the costs?

    2.Cost benefit data.

    		 

    X

    X

    		 

    X

    X

    		   

    3.Are community members satisfied with the participants and the service they provide ?

    3.Satisfaction ratings from beneficiaries and community members.

    X

    		       

    X

    		   

    *Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

    • Survey/Scale: i.e., member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys
    • Structured Interview: i.e., interview with key participants
    • Staff Report/Log: i.e. monitoring system, goal attainment report
    • Direct Observation: This refers to the direct observation of behavior by researchers .
    • Archival Records: i.e., community-level indicators of impact

    **Here are explanations of these terms:

    • Case Study: an in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment--good for answering "how" or "why" questions
    • Pre-/Posttest Control Group: monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made
    • Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative


    Example 3: Timeline for Evaluation Activities

    This is the timeline for an evaluation being done by Kansas City's Promise on their initiative to promote community and personal caring for youth:

     

    1998

    1999

    2000

    2001

    Monitoring System

    M

     A 

    M

     J 

     J 

    A

    S

    O

    N

    D

    J

    F

    M

    A

     M 

     J 

     J 

    A

    S

    O

    N

    D

    J

    F

    M

     A 

     M 

     J 

     J 

    A

    S

    O

    N

    D

    J

    F

    M

    		 

     Establish system

    [ -

    -

    - ]

    		                                                                     

     Train participants/staff in data entry

    [ -

    -

    - ]

    		                                                                     

     Maintain system
     (data collection & reporting)

    [ -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    - ]

    		 

    Critical Events

    		                                                                           

     Identify key people

    		         

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]

    		           

     Conduct interviews

    		             

    [ -

    - ]

    		                   

    [ -

    - ]

    		                   

    [ -

    - ]

    		         

     Report results

    		                   

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]

    		               

    [ -

    -

    - ]

    		   

    Community-Level Indicators

    		                                                                           

     Identify local indicators

    		               

    [ -

    -

    -

    - ]

    		                                                   

     Identify comparison site

    		               

    [ -

    -

    -

    - ]

    		                                                   

     Secure data

    		                     

    [ -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    - ]

    		         

     Report results

    		               

    [ -

    -

    - ]

    		                     

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]

    		 

    Other Indicators (e.g. behavioral surveys, data observations)

    		                                                                           

     Identify

    		           

    [ -

    -

    -

    -

    -

    -

    - ]

    		                                                 

     Secure data

    		                   

    [ -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    -

    - ]

    		       

     Report results

    		                   

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]

    		                 

    [ -

    -

    - ]