Search form

Ask an Advisor Answers

Question:
Hello,
I am a doctoral student who is using your resources to guide a community based assessment survey. Do you have any validity or reliability stats for Community Tool Box as a whole or from any of the examples and resources you offer?
Thanks so much!
Answer:

Hi Amy,
Thank you for visiting Ask an Advisor @ the Community Tool Box, I hope we can provide you with useful and directly applicable information to get your on your way! Okay, what I think you're asking is whether we have information about the reliability or validity of community assessment surveys, correct? Well, I did a little digging around and sorry to say I could not come up with any reliability or validity statistics within any of our chapters, toolkits, or other resources. If I had to hazard a guess, you've followed this thread: Home » Table of Contents » Community Assessment » Chapter 3. Assessing Community Needs and Resources » Section 13. Conducting Surveys to inform your work? If so, then you probably already know, the only survey example we have is the CDC's Youth Risk Behavioral Survey (or YRBS) found at http://www.cdc.gov/HealthyYouth/yrbs/, but no reliability or validity statistics are provided that I could find. All that said, as a doctoral student you've likely taken a research methodology sequence that might have included survey research methods and statistics, yes? Well, if not you will. Now Amy, I'm still not totally clear whether you're looking for reliable and valid community assessment surveys that already exist or you are designing your own survey and want to know how to determine whether it's reliable and valid, but based on your question I suspect it's the latter? If it's the former then a Google Scholar search or using your own institution's on-line journal search engines (e.g., Web of Science or PsychInfo) will uncover a treasure-trove of surveys which are reliable, valid, or both that you can adapt or modify to suit your purposes (this is not necessarily a bad way to go if you're so inclined). If it's the latter, then I suggest working with a survey research expert who can provide guidance if you don't feel comfortable designing and implementing the survey yourself. Believe me when I tell you from my own experience, I had a killer survey research methods course when I did my doctoral work and feel pretty competent in the totality of the survey research enterprise, but I still consult with experts because they can see things from an objective perspective that we cannot; there is absolutely no shame in doing so as it is so easy to completely run a potentially good survey off the rails (in a manner of speaking). Because your question is really about survey/scale reliability and validity a very brief primer is in order. If you already know about what I'm about to say then you can skip it, no harm done. Regarding reliability, if you are designing your own survey you would need to have a sufficient amount of data already collected to calculate Chronbach's alpha. Many people will pilot test a survey and then do the reliability calculation (easily done in SPSS). If reliability is low, say <.75 and the sample sufficiently large then a factor analysis would be done to get rid of like items, which would increase internal consistency, hence reliability (this is also important if in future analyses you plan on constructing regression models or doing other types of modeling). However, you have to have upwards of 200 survey participants with nearly complete data to run the factor analysis.  Even with a sample size of, say 100ish you can calculate the reliability coefficient to at least give you a rough estimate of internal consistency.  Now validity is another game altogether and merits a much longer and more sophisticated discussion.  In essence, creating a valid scale or survey requires repeated administrations over a number of years and a lot of data from sufficiently large samples; this is not something you are likely going to want to do because it is time consuming and very labor intensive.  In some ways I don't think you need to worry about either reliability or validity because you are assessing needs in a community not measuring a psycho/socio/emotional construct, you're merely trying to determine what concerns people in a community have regarding the conditions in which they live and work.  So, in sum Amy: 1) find community assessment surveys that are reliable and valid and adapt them to your own needs using different academic search engines; 2) consult with a survey research expert to help you construct and implement your survey; and 3) unless you're measuring a psychological construct then reliability and validity are not as much a concern as you might think.  Hope all that makes sense.  Let me just say in final analysis that designing and constructing surveys requires a deft hand and a keen understanding of the cognitive processes required of survey respondents so it is an enterprise that is fraught with peril; in fact there is a whole scientific base behind survey construction and administration, and scholars have devoted a life-time of academic research to this niche within the larger social science research field.  If you haven't already, read Don Dillman's work, he's the "go-to guy" in survey research.  Again, Amy thank you for your question and we hope we've given you plenty of information to consider and that it's useful to you; if not, please feel free to get in touch with us again and we'll work together to find a better answer.  Be well and take care, and remember, you're never alone!
Ask an Advisor
          

Question Date: Fri, 04/22/2016