Tool: Team Data Collection - How to Do It Well
This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007. We have included her initial e-mail, and then added the summarized reponses to each of her questions.
From: Michelle Garred
Subject: Team data collection: how to do it well?
I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines. Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.
Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well.
Among the many questions are:
- What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
- How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
- How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
- In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
- How would you cite team interviews in a Ph.D. thesis?
- What are the key resources I should review?
What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
- Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.
- Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.
- More personalities and activities to coordinate means more opportunities for things to go wrong.
- Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.
- Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.
Pitfalls can be addressed through training:
- Need for consistent, systematic training of data collectors.
- Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.
- Conduct interviewing practice sessions before beginning actual interviews.
- During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.
- Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.
- Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?
- Demonstrated skills before graduation from training phase
And through regular team meetings:
- Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along.
- Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.
- Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.
- If not together on site, consider on-line conferencing.
How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
Keep in mind that we will have a team of 5 people conducting semi-structured interviews and we will probably not use audio recording.
- Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.
- Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.
- Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked. Here's one example of a card storming.
- If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.
How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
Again, this can be addressed through team training and regular meetings . . .
- Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.
- Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.
- Revisit this issue during regular team meetings during the data collection phase.
And through wise design of the note-taking protocols:
- Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.
- Additionally, you might have team members write up a brief summary within 24 hours of completing the interview.
- Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.
In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
I understand that team data collection is common in action research, but less common in a Ph.D. thesis. It would help me to hear some different opinions on this issue.
- For quantitative data, one might set a quota of random 10% of data reviewed for QA.
- I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.
- As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
- Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
- Zuber-Skerritt, Ortrun, and Perry, Chad (2002) Action research within organisations and university thesis writing. The Learning Organization, 9(4), 171-179.
How would you cite team interviews in a Ph.D. thesis?
- In table format within the document.
- I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
- I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
- When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
- Consider in advance whether you will use the participants' real names or pseudonyms.
- See Madeline Church's doctoral thesis, Creating an Uncompromised place to belong - why do I find myself in networks? from the living theory section of Action Research.
- The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews. You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006). "Finding common ground in team-based qualitative research using the convergent interviewing method." Qualitative Health Research, 16(8), 1145-1157.
- There's a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006). Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.
- A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about. You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.