Section 13. Conducting Surveys

Learn the basics of conducting surveys, including survey creation, distribution and collection, and how to utilize survey data.

 

• What are surveys?

• Why should you conduct a survey?

• When should you conduct a survey?

• How do you prepare a survey?

• How do you distribute your survey?

• How do you analyze and compile the results of your survey?

When you want somebody's opinion, you ask for it. Right? That's easy enough when you're just dealing with one or a few people. But what if you want to know the opinion of an entire town or an entire population? Getting an answer out of everyone in your town or every member of a particular group is nearly impossible. So how do you get an idea of what these folks think? You use a survey.

Conducting surveys can be done very simply, or it can be very complicated, depending on how much you want to ask on the survey and the number of people to whom it is administered. This section will mainly focus on doing surveys on a fairly small local scale, and we will give you some ideas about where to find information should you need to do a survey on a larger scale.

What are surveys?

A survey is a way of collecting information that you hope represents the views of the whole community or group in which you are interested.

There are three main ways of going about this:

1. Case study surveys, which collect information from a part of a group or community, without trying to choose them for overall representation of the larger population. You may need to conduct several of these before you get a sense of how the larger community would respond to your survey. Case study surveys only provide specific information about the community studied.
2. Sampled surveys, which are the type we'll be focusing on in this section, ask a sample portion of a group to answer your questions. If done well, the results for the sample will reflect the results you would have gotten by surveying the entire group. For example, let's say you want to know what percentage of people in your county would make use of an adult literacy program. Getting every person in a county with 10,000 people to fill out a survey would be a huge task. Instead you decide to survey a sample of 500 people and find out what they think. For the sample to accurately represent the larger group, it must be carefully chosen. We'll speak to that later in this section.
3. Census surveys, in which you give your survey questionnaire to every member of the population you want to learn about. This will give you the most accurate information about the group, but it may not be very practical for large groups. A census is best done with smaller groups -- all of the clients of a particular agency, for example, as opposed to all of the citizens of a city.

Surveys are usually written, although sometimes the surveyor reads the questions aloud and writes down the answers for another person; they can be distributed by mail, fax, e-mail, through a web page, or the questions can be asked over the phone or in person.

Surveys collect information in as uniform a manner as possible -- asking each respondent the same questions in the same way so as to insure that the answers are most influenced by the respondents' experiences, not due to how the interviewer words the questions.

Why should you conduct a survey?

You can collect information about the behaviors, needs, and opinions using surveys. Surveys can be used to find out attitudes and reactions, to measure client satisfaction, to gauge opinions about various issues, and to add credibility to your research. Surveys are a primary source of information -- that is, you directly ask someone for a response to a question, rather than using any secondary sources like written records.

You can use surveys to measure ideas or opinions about community issues related to your initiative. For example, you may want to know how many people use your services, what users think about your services, what new users expect from your services, and whether users are satisfied with what you provide.

Deciding whether to conduct a survey

There are advantages in doing surveys, but you should consider whether a survey will be the best way of obtaining the information you need. Even though surveys are a useful method of gathering information, they are not the only way. You will need to decide whether a survey will produce the information you need. The information you need may be obtained through other means, such as informal unstructured conversation that takes place in the course of another activity; census figures; meeting with people in the community; interviews; or observation.

When should you conduct a survey?

A survey may be your best choice when:

• You need a quick and efficient way of getting information
• You need to reach a large number of people
• You need statistically valid information about a large number of people
• The information you need isn't readily available through other means

Written surveys: Pros and Cons

Advantages of written surveys:

• Large numbers of people can give their input
• Low cost
• People can respond at their convenience
• Avoids interviewer bias
• Provides a written record
• Easy to list or tabulate responses
• Wide range of respondents
• No training needed as with interviewing

Disadvantages of written surveys:

• Often has low return rate
• Limited alternative expression of respondent's reaction
• Depends on the selected sample
• May not truly represent of the whole group
• Respondent may skip sections

If you have decided that what you need is a large-scale, formal survey, hiring someone to do it for you or working with local colleagues or a nearby university may be your best bet. If you're going to do it on your own, keep in mind that some people you present your report to may not give much credit to a survey you did on your own.

How do you prepare a survey?

Decide on the purpose of the survey.

If you have decided to do a survey, you must first be sure exactly why you're doing it. What questions do you want to answer? Is it to get a general idea of the demographics of your area? To find out what people think about a particular issue or idea? Or is there another reason you're considering a survey?

In any case, you will need to keep the purpose of the survey in mind throughout the process, as it will influence the choice of questions, the survey population, and even the way the survey is delivered (e.g., a computer-savvy population can be surveyed over the Internet; a population that is largely illiterate shouldn't be asked to take a written survey, and so forth).

Example: 1997 Youth Risk Behavioral Survey purpose

The Youth Risk Behavioral Survey (or YRBS) is done annually by the Centers for Disease Control to identify behaviors that pose health risks among young people in America. We will be using the 1997 and 1999 YRBS for examples in this section.

The CDC decided its purpose in this survey was to track the health risk behaviors that cause the most deaths among youth. Also, many of those behaviors are included in the survey because they begin in youth and continue into adulthood, having significant impact on adult health later on. Here are some of the behaviors the YRBS attempts to measure:

• Behaviors that contribute to unintentional and intentional injuries (like not using a safety belt when driving)
• Tobacco use
• Alcohol and other drug use
• Sexual behaviors that contribute to unintended pregnancy and sexually transmitted diseases, including HIV
• Unhealthy dietary behaviors
• Physical inactivity

Decide whom you will survey.

The next step is finding out who has the answers to your question or questions. In other words, it's time for you to determine your audience -- the people who can best answer the questions your initiative needs to ask. Who will you survey? Is it the general public? The current program beneficiaries? People in a specific neighborhood or segment of the community? Potential members?

Sampling

Almost all surveys rely on sampling -- that is, identifying a section of your population that satisfies the characteristics you're trying to survey, rather than trying to do a census.

To have a truly representative sample, you must be sure that every member of the group you want to survey has an equal chance of being in the sample, and/or you must have a fairly large sample. It's important to make sure that the sample size you choose is adequate and not excessively large or small. If too large, it may be impossible to survey everybody effectively and within your budget; if too small, your credibility may suffer. A general rule to keep in mind is that the larger the sample size, the more accurate a reflection of the whole it will be.

You can figure out how big your sample should be by using a sample size calculator, such as:

• The Survey System's Sample Size Calculator allows you to decide whether you want to calculate for 95% or 99% confidence level (the statistical term for the amount of certainty you have about the accuracy of your results).
• UCLA's Sample Size Calculator from the online statistics textbook is a bit more advanced.

Sample Design Issues

You might also need to give some thought to the design of your sample, especially if you are hoping to get representative responses from two or more groups. For example, let's say you are doing a survey on youth violence and you want to get responses from youth, parents, and educators; this means that you'll need to come up with separate population counts for each of these groups and then select a sample from each. The samples should be large enough to represent the group it is drawn from, but the sample sizes should be proportional to the groups they represent.

For example, you might design a sample that comes out like this:

	Youth	Parents	Educators
Population	650	200	500
Sample	65	20	50

Sampling can be a complex topic; before you start, you might want to learn some of the basic terminology and concepts from resources such as Sampling Terminology and Survey Sample Sizes and Margins of Error.

Potential pitfalls

Sampling is a challenge to conducting good surveys, but there are other pitfalls. For example, when people volunteer to respond to a survey, we say they are self-selected. These people may have a special interest in answering your survey, so their answers may not be truly representative of the group you're interested in. There are ways of dealing with self-selected audiences, such as only using a random selection of their surveys when only self-selection is involved.

For example, if you get back 300 completed surveys, you might decide to only use every third one in order to randomize the results.

Decide what method you will use to collect your survey data.

Will your survey be written or oral? Is there going to be a number where people can call to register their results? Are you going to have a post office box to which completed surveys should be mailed? You need to decide whether it's going to be administered by people known to the audience and whether it will be done in person, by phone, or by mail. Remember that the more personal you make it, the higher the return rate will be. Surveys that are delivered cold have a return rate of only two to three percent, unless they're on a very hot topic for the community you're surveying.

Keep in mind whom you want to survey. Does your public feel more comfortable writing or speaking? Will it be efficient to leave surveys somewhere for people to pick up at their will, or should you do something to make sure they get one? If your survey is to be administered orally, will people feel honored or annoyed about being asked for their opinions?

Mailed questionnaires are a very useful tool in your information-gathering bag of tools. It's a much cheaper alternative to other types of information gathering and it allows you to get information from many people across long distances without paying extremely high phone bills. If you're considering doing a mailed survey, be sure to check with your local post office for information on mailing regulations, bulk mail rates, and so on.

Some advantages of mailed questionnaires are:

• The respondent can fill out the survey at his or her convenience -- it can be filled out whenever the respondent has time.
• You can make it anonymous, which is much more comfortable for some respondents.
• All respondents will have read the same questions, eliminating any interviewer bias.
• The respondent will have time to check his or her records before answering -- if he or she needs to verify information, he or she will have the chance to be accurate.

Some disadvantages of mailed questionnaires are:

• They're not very flexible; there is no interviewer present to probe for answers, so you can only read what the respondent has written, with no opportunity to look at facial expressions or body language.
• The return rate is generally low
• Respondents may leave answers blank
• You can't control when respondents will send the survey back
• You may not be able to tell the difference between those who simply didn't return the survey and those for whom you had an incorrect address.

How long should your survey be?

When determining the length of your survey, remember that less is more. The longer it is, the less likely it is that people will take the time to do it. People get bored with long surveys, and usually won't even bother to look at a survey that is more than a page and a half long. Also, requiring long answers may lose your audience. Through editing and condensing, you should try to keep your survey down to one page.

What it is you want to know and the method of survey (e.g., phone survey, mailed survey) will also influence the length of your survey. Phone surveys, for example, can take a little longer to complete.

Once you've decided on your method, you can go on to write your questions. We'll talk in more detail about distributing your survey later on.

Example: 1997 Youth Risk Behavioral Survey sampling

The 1997 YRBS used a type of sampling called cluster sampling. In cluster sampling, the entire population is divided into groups, or clusters, and a random sample of these clusters are selected. For example, age group or geographical location determined the YRBS's clusters. All observations in the selected clusters are included in the sample. This technique is used in large-scale surveys where it may be more convenient to sample clusters than to do a pure random sample.

Write your questions.

When preparing the questions, bear in mind that they can take many forms. Questions might be:

• Open-ended: Designed to prompt the respondent to provide you with more than just one or two word responses. These are often "how" or "why" questions.

For example: "Why is it important to use condoms?" These questions are used when you want to find out what leads people to specific behaviors, what their attitudes are towards different things, or how much they know about a given topic; they provide good anecdotal evidence. The drawback to using open-ended questions is that it's hard to compile their results.

• Closed-ended (also sometimes referred to as forced choice questions): Specific questions that prompt yes or no answers.

For example: "Do you use condoms?" These are used when the information you need is fairly clear-cut, i.e., if you need to know whether people use a particular service or have ever heard of a specific local resource.

• Multiple choice: Allow the respondent to select one answer from a few possible choices.

For example: "When I have sex, I use condoms... a) every time, b) most times, c) sometimes, d) rarely, e) never." These allow you to find out more detailed information than closed-ended questions, and the results can be compiled more easily than open-ended questions.

• Likert scale: Each respondent is asked to rate items on a response scale. For instance, they could rate each item on a 1-to-5 response scale where:

1 = strongly disagree

2 = disagree

3 = undecided

4 = agree

5 = strongly agree

If you want to weed out neutral and undecided responses you can use an even-numbered scale with no middle "neutral" or "undecided" choice. In this situation, the respondent is forced to decide whether he or she leans more towards the "agree" or "disagree" end of the scale for each item. The final score for the respondent on the scale might be the sum of his or her ratings for all of the items.

Example: Using the Likert scale

Here are a few sample survey questions in Likert scale format, done without a neutral category:

Please check the answer indicating your reaction to the questions listed below.

• Strongly Disagree
• Disagree
• Agree
• Strongly Agree
• Violent crime is a significant problem in my neighborhood
• The police have done enough to prevent crime in my neighborhood.
• If a citizens watch program were implemented in my neighborhood, I would participate in it.
• I would be supportive of organized activities for youth in my neighborhood.

The questions you ask depend on the audience you're trying to reach and the information you're trying to obtain.

For example, for demographic information (e.g., questions that determine where people are from, their ages, and their incomes), you should make the survey all check-offs, yes/no questions, and fill-in-the-blank questions so that it?s as easy as possible to complete.

Creating surveys people will answer with the Total Design Method

Low response rates are a major problem with surveys; it's common for the response rate to be as low as 30%. One way of avoiding low response rate is to use the Total Design Method, which was developed by Don Dillman of Washington State University. Dillman's method has been shown to yield an average return rate of 73%.

• Mailed questionnaires should be printed on standard letter paper (8.5 x 11"), then folded in half into a booklet. This size of envelope is less likely to be viewed as advertisement or "junk" mail by the recipient, so more people will open your survey.
• There should be no questions on the front or back of the folded booklet.
• The first question should be directly related to the overall topic of the survey, and it should be something that is easy to answer. Any questions that may be threatening to the reader should appear later in the survey, but not grouped together. Demographic questions should come towards the end; having them at the beginning often puts people off and prevents them from completing the survey at all, but they will be more likely to complete them if they are asked after responding to other questions.
• In layout, avoid cramming too much type onto a single page. It's better to use more pages with a good amount of white space than to try to save on paper by crowding the pages, because overly-dense type is intimidating to a potential survey participant. You should also make sure you don't break any questions up over a page break -- the entire question and its possible answers should appear on the same page.
• Your questionnaire should be no more than 125 questions or 12 pages long -- anything longer is going to reduce your response rate.
• Including a well-written cover letter is extremely important. It needs to be clear about what you're looking for, why you're looking for it, what member of the household should complete the survey, and what will be done with the results. For instance, if you're doing a literacy program survey, you may want to explain that the answers will help determine whether the community might need an adult education program and what kind of program it might need, and that the results of the survey will be presented to possible funders. Your cover letter should be individually typed or laser printed and signed personally with a blue ballpoint pen (survey participants pay more attention to real letters with real signatures).
• Typing the recipient's name directly on the envelope, instead of using mailing labels, will bring a higher response rate. Using first class postage -- especially commemorative, colorful stamps -- will increase your response rate even more.
• Follow up with those who haven't responded after a week with a postcard, politely reminding them about the survey. After the second week, send a new cover letter and questionnaire to those who have not yet responded. After the fourth week, send yet another questionnaire, this time by certified mail, along with a letter reminding the recipient that you haven't yet received his or her survey and that his or her response is very important.

Questions should be worded carefully in order to yield exactly the information you're looking for.

To make sure your survey works the way you want it to, try it out on a few members of the population you're aiming at before you actually distribute it.

Guidelines for writing your survey questions:

• Place easier questions first
• Address sensitive issues as discreetly and sensitively as possible
• Avoid words that provoke bias or emotional responses
• Use a logical order and place similar questions together

Example: 1999 Youth Risk Behavioral Survey questions

Here are a few examples of questions from the 1999 YRBS.

• During the past 30 days, how many times did you ride in a car or other vehicle driven by someone who had been drinking alcohol?

A. 0 times
B. 1 time
C. 2 or 3 times
D. 4 or 5 times
E. 6 or more times

• During the past 12 months, did you ever seriously consider attempting suicide?

A. Yes
B. No

• How old were you when you smoked a whole cigarette for the first time?

A. I have never smoked a whole cigarette
B. 8 years old or younger
C. 9 or 10 years old
D. 11 or 12 years old
E. 13 or 14 years old
F. 15 or 16 years old
G. 17 years old or older

• During your life, how many times have you sniffed glue, breathed the contents of aerosol spray cans, or inhaled any paints or sprays to get high?

A. 0 times
B. 1 or 2 times
C. 3 to 9 times
D. 10 to 19 times
E. 20 to 39 times
F. 40 or more times

• During your life, with how many people have you had sexual intercourse?

A. I have never had sexual intercourse
B. 1 person
C. 2 people
D. 3 people
E. 4 people
F. 5 people
G. 6 or more people

How do you distribute your survey?

There are several strategies for distributing surveys. We'll talk about the most common one -- direct mail -- in the most detail, but there are many methods to choose from and there is no one perfect method. You may want to use a combination of methods.

Here are a few thoughts to help you decide on your method:

• Self-administered questionnaires are better than interviews when you're dealing with respondents who can read and write and the questions you're asking don't require any visual aids like charts, graphs, etc. that might need explanation.
• Phone surveys work well in the place of self-administered questionnaires if at least 80% of the population you're working with have phones in their homes. They also work better if the questions are of a nature that respondents might be uncomfortable or embarrassed to give their answers to an interviewer.

For example, if you are doing a survey on sexual risk behaviors, people may be uneasy telling an interviewer how many partners they've had or other such details.

• Drop boxes work best if you have limited human resources or if you are in a place where the mail and phone systems aren't adequate.

Direct mail

Direct mailing your survey to people whose addresses are known is the most common strategy. Distributing a survey by mail has a high percentage of non-responders (you're lucky if 30% respond, although it tends to be higher in small communities), but it's a lot easier than many other methods and takes less staff hours.

• Gather the items you'll need to do a direct mailing:
  • Mailing labels or a mailing list: If you're mailing the survey to everyone in town, the city's billing lists for water bills might be a good source of a mailing list. The mailing list of relevant agencies can also be useful. Good resources might be the public health department, the Salvation Army, relevant United Way agencies, emergency medical services, or companies that develop phone books. If you're using an agency's mailing list, be sure to get permission from the agency's director before doing the mailing. Give the director a sample survey and a copy of the cover letter to review and invite him or her to suggest any changes that might further protect his or her clients.
  • Two business envelopes and two stamps for each participant: One set to send the survey to the participant and one for it to be returned in. The return envelope should be pre-stamped and pre-addressed.
  • One copy of the survey, demographics sheet, and cover letter for each participant.
• Complete the cover letter. A sample cover letter you may want to use as a guide appears in the Examples section.
• Make enough copies of the survey, demographic sheet, and cover letter for each survey recipient.
• Prepare the two business-size envelopes for each person. One should have the agency's return address and a mailing label for the survey participant; the other should have the agency's address listed as both the mailing and the return addresses. Stamp both envelopes.
• Stuff the envelopes that have the recipient's mailing address with all the survey materials -- the survey, the demographic sheet, the cover letter, and the return envelope.
• If you want to track the surveys in any way -- trying to see what sort of answers you get from different parts of town, for example -- you may wish to code the envelopes in some way. One way you can do this is by numbering each return envelope and keeping a copy of the mailing list with matching numbers -- for example, if John Doe at 123 Main Street is assigned number 007, then the number 007 will also be on his return envelope. Another option is to color code the surveys by zip code.
• Mail them out! Try to get a bulk rate to reduce costs.
• If less than 10% of the distributed surveys are returned, try one or more of the following strategies:
  • Send a reminder to all or a random sample of people on the mailing list.
  • Contact the local newspaper and request an article on the survey, submit a letter to the editor about it, or publish an announcement about the survey. This is something you should do before you send out the survey.
  • Contact radio stations to run announcements inviting people to take part in the survey.
  • Invite citizens to participate in the survey through announcements in local agency newsletters, consumer group meetings, and public community events.
  • Post announcements of the survey in public places, like the library or grocery stores.

Interviews and phone surveys

For those who have difficulty reading or using printed materials, or for surveys that require more in-depth answers, interviews might be the most appropriate thing for you to do. Phone surveys work similarly to face-to-face interviews, so we've grouped these two methods together.

• Put together a team of interviewers. The people you choose should be able to answer any questions respondents might have, and if necessary they should be people who can handle meeting diverse respondents. People who work in the social sciences often have interviewing experience.
• Train the interviewers to act as a team. They should all be given the same information about the survey, its purpose, and your organization or initiative to make sure that the information they pass on to respondents is uniform.
• For a phone survey, your sample can be as simple as every fifth phone number in the white pages of your local phone book, or you may need to work with a survey consultant to get a phone list of a more specific sample group.
• Phone interviewers should be polite, call during reasonable hours (not at meal time and not too late at night or early in the morning, etc.), and they should all be consistently asking the same questions.

Drop boxes

Agencies that have relatively frequent contact with clients -- such as once a month -- you may find that setting up a drop box in their offices are a good source point for distributing surveys. This may also be a good option for agencies that have an incomplete mailing list. It can also be a good way to contact clients of other agencies who have little contact with your group or agency. However, if you use this method of distributing surveys, consider using it along with at least one other method of distribution, because only those already using the services can respond.

Media distribution

For general distribution, publishing a survey in the local paper or attaching a survey to your newsletter might be a good idea.

Convenience sampling

Taking surveys in a public place -- setting up a booth or table in the parking lot at a local discount store, on the sidewalk in the shopping district, etc. -- provides an opportunity to get some exposure for your organization.

Group administration

If your group or organization tends to have large group gatherings, providing surveys to everyone who attends a particular gathering might be a really efficient way for you to gather information. Examples of gatherings where you might want to distribute your survey would include: immunization clinics, commodity food distribution sites, health fairs, and meal sites for older adults. If you want to give your survey out at some sort of group meeting or gathering, get the group's director to put you on the agenda. At the meeting, introduce yourself and explain the purpose of the survey. Then distribute the survey, answer any questions, and collect completed surveys. Don't forget to thank everyone for their participation!

Using multiple methods of distribution

You can combine or adapt two or more of the above methods to suit your own purposes, if you'd like. If more than one method is used, each survey should include instructions that each citizen should complete only one survey. So, for example, if you're having people complete surveys at a booth at the county fair, they should not complete the survey if they've already completed one that came in the mail to their homes.

Collecting the surveys

Soon after the surveys are distributed, some of them will begin to arrive at the sponsoring organization. Here are the steps you should take to collect your surveys:

• Gather incoming surveys collected at participating sites. A representative of your organization should collect incoming surveys as they arrive in the mail or your drop box. He or she should also call or stop by collection sites from time to time to pick up any surveys that have been dropped off.
• Review returned surveys, checking for any that are incomplete. If any surveys were returned for having an improper mailing address, try to find the correct address and mail it out again, if you can.
• Secure a larger return, if necessary. This may mean distributing surveys again or expanding your sample size.

Example: Administering and collecting the 1997 Youth Risk Behavioral Survey

The CDC wanted to do everything it could to protect the students' privacy and insure that questions would be answered honestly while completing the YRBS. In order for the survey to be administered voluntarily and anonymously, it was done in a self -administered written questionnaire containing 84 multiple-choice questions. Before the surveys were administered, parental permission was obtained through whatever methods those local schools used. Students recorded their responses to the questionnaires on computer-scannable answer sheets, further allowing for anonymity.

How do you analyze and compile the results of your survey?

Now that you've gathered the completed surveys, you'll need to figure out the results. Sometimes all you have to do is tabulate the results -- that is, add them up and display in a table.

For instance, if 100 questionnaires were returned in a survey about problems in the neighborhood, you just need to count the answers. Let's say that there was a question asking what people felt was the biggest challenge facing the neighborhood; 70 people mentioned law enforcement, 10 cited transportation, 15 marked potholes, and 5 said noise. The result in cases like this is clear.

However, analysis can be far more complicated than that. If you're looking,

for instance, at how people feel about a service or problem, you may end up with a lot of answers to open-ended questions that are apparently unrelated. In this case, you will need to try to find patterns.

Once you've done that, what do these numbers mean? Well, you will need to look at the overall survey to see how each percentage compares to the others.

For example, what questions had the highest proportions of similar responses?

We suggest that you write up a brief report -- one page is sufficient -- summarizing the results of the survey. In your report, look for any patterns -- do people in a particular part of town feel more strongly about a particular issue than those in other areas?

Share this information with your staff. Get their feedback and discuss whether any further surveying needs to be done before completing.

Now that you've figured out what the results mean, you need to decide what to do with them. To whom are you going to communicate them, and how? In case of a community initiative, the results should be made public as soon as possible so that members in the community and community leaders can be made aware of a problem or potential problem and start working to solve it. If other similar surveys have done in the same area, you may want to compare your results with the other surveys' results.

An organization conducting a survey about its' services might want to use results to provide a better service or to change a current policy to a more efficient one. In a situation where funding is at stake, the results would need to go to the funder to convince the funder of the need for new or continued support. The results could also be used by the organization itself to determine where and what kinds of services are needed.

The 2003 Youth Risk Behavior Survey results will give you an idea of what a survey result report looks like.

In Summary

A well-executed survey can provide your initiative with a wealth of information about your constituents and their needs. We hope this section has given you the tools you need to conduct surveys that are effective and that give you the information you need to serve your constituents better!