Table of Contents >
   Part J. Evaluating Community Programs and Initiatives
      Chapter 36. Introduction to Evaluation >
         Section 6. Participatory Evaluation >
             Tools & Checklists - A checklist that summarizes the major points contained in the section. >

Participatory Evaluation

  

Tools & Checklists

 Contributed by Phil Rabinowitz Edited by

Checklist

Tool #1: Team Data Collection: How to Do It Well?

 

Checklist

Here you will find a checklist summarizing the important points of the section.

What is participatory evaluation?
__ Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work.

Why would (and why wouldn't) you use participatory evaluation?
Advantages:
__ It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects.
__ It can get you information you wouldn't get otherwise.
__ It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff.
__ It can tell you why something does or doesn't work.
__ It results in a more effective project.
__ It empowers stakeholders.
__ It can provide a voice for those who are often not heard.
__ It teaches skills that can be used in employment and other areas of life.
__ It bolsters self-confidence and self-esteem in those who may have little of either.
__ It demonstrates to people ways in which they can take more control of their lives.
__ It encourages stakeholder ownership of the project.
__ It can spark creativity in everyone involved.
__ It encourages working collaboratively.
__ It fits into a larger participatory effort.
Disadvantages:
__ It takes more time than conventional process.
__ It takes the establishment of trust among all participants in the process.
__ You have to make sure that everyone's involved, not just "leaders" of various groups.
__ You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills.
__ You have to get buy-in and commitment from participants.
__ People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation.                                                                                                                                                                                                                                                                                           
__ You may have to be creative about how you get, record, and report information.
__ Funders and policy makers may not understand or believe in participatory evaluation.

When would you use participatory evaluation?
__ When you're already committed to a participatory process for your project.
__ When you have the time, or when results are more important than time.
__ When you can convince funders that it's a good idea.
__ When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of.
__ When you need information that it will be difficult for anyone outside the community or population to get.
__ When part of the goal of the project is to empower participants and help them develop transferable skills.
__ When you want to bring the community or population together.

Who should be involved in participatory evaluation?
All stakeholders, including:
__ Participants or beneficiaries.
__ Project line staff and/or volunteers.
__ Administrators.
__ Outside evaluators, if they're involved. 
__ Community officials.
__ Others whose lives are affected by the project.

How do you conduct a participatory evaluation?
__ Recruit stakeholders as participant evaluators.
__ Train evaluators.
__ Name and frame the issue.
__ Develop a theory of practice to address it.
__ Determine the evaluation questions.
__ Collect information.
__ Analyze the information.
__ Use your analysis to celebrate what worked and adjust the rest to improve the project.
__ Stick with it indefinitely.

 

Tool #1: Team Data Collection: How to Do It Well?

This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007.  The text in bold represents the original inquiry. All other text represents the summarized responses.

From: Michelle Garred <mgarred5@hotmail.com>
Subject: Team data collection: how to do it well?

Dear colleagues,

I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines.  Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.

Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well. Among the many questions are . . .

1. What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?


Advantages:

  • Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.

  • Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.

Pitfalls:

  • More personalities and activities to coordinate means more opportunities for things to go wrong.

  • Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.

  • Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.

The pitfalls can be address through training . . .

  • Need for consistent, systematic training of data collectors.

  • Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.

  • Conduct interviewing practice sessions before beginning actual interviews.

  • During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.

  • Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.

  • Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?

  • Demonstrated skills before graduation from training phase

. . . And through regular team meetings.

  • Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along. 

  • Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.

  • Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.

  • If not together on site, consider on-line conferencing.

2. We will have a team of 5 people conducting semi-structured interviews. We will probably not use audio recording. Therefore how can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the DEPTH and CONSISTENCY of data?

  • Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.

  • Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.

  • Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked.  One example of card storming can be found at http://www.ncrel.org/sdrs/areas/issues/educatrs/profdevl/pd2reach.htm.

  • If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.

3. How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill? 

Again, this can be addressed through team training and regular meetings . . .

  • Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.

  • Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.

  • Revisit this issue during regular team meetings during the data collection phase.

. . . And through wise design of the note-taking protocols:

  • Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.

  • Additionally, you might have team members write up a brief summary within 24 hours of completing the interview. 

  • Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.

4. I understand that team data collection is common in action research, but less common in a Ph.D. thesis. Thus I wonder, in my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me? It would help me to hear some different opinions on this issue.

  • For quantitative data, one might set a quota of random 10% of data reviewed for QA.

  • I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.

  • As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
  • Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
    • Zuber-Skerritt, Ortrun, and Perry, Chad (2002)  Action research within organisations and university thesis writing.  The Learning Organization, 9(4), 171-179.

5. How would you cite team interviews in a PhD thesis?

  • In table format within the document.
  • I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
  • I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
  • When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
  • Consider in advance whether you will use the participants' real names or pseudonyms.

Other key resources:

1. See Madeline Church's doctoral thesis from the living theory section of http://www.actionresearch.net at:
http://people.bath.ac.uk/edsajw/church.shtml.

2. The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews.  You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006)  Finding common ground in team-based qualitative research using the convergent interviewing method. Qualitative Health Research, 16(8), 1145-1157.
 
There's a description of it on the web at http://www.uq.net.au/action_research/arp/iview.html and a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006)  Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.

3. A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about.  You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.